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Abstract 

This  pilot  research  study  was  intended  to  identify 
factors  which  affect  the  length  of  the  source  selection 
process  and  to  suggest  changes  to  make  the  process  more 
efficient.  To  accomplish  this  objective,  interviews 
were  conducted  with  Aeronautical  Systems  Division  (ASD) 
contracting  personnel  who  had  recently  participated  in 
source  selections. 

The  study  identified  several  factors  which  appear 


to  bo  significant  contributors  to  the  length  of  the 
source  selection  process.  These  factors  are: 


1) 

lack  of  understanding, 
experience,  and  training; 

2) 

lack  of  teamwork; 

3) 

lack  of  quality  control; 

4) 

lack  of  written  guidance; 

5) 

effects  of  ASD  streamlining 
initiatives ; 

6) 

excessive  oversight; 

7) 

unneccessary  or  excessive 
requirements ; 

8) 

manpower  constraints;  and, 

9) 

pol i tics . 

The  recommendations  of  this  study  are 

intended  to 

limit  source 

selection  activities  to  those 

essential  to 

the  integrity 

of  the  process,  remove  extraneous 

participants 

from  the  process,  and  ensure 

that  actions 

are  completed  correctly  the  first  time.  The  most 


ix 


significant  recommendations  related  to  specific 
activities  include: 

1)  changing  public  law/FAR  requirements 
to  allow  screening  of  CICA  synposig 
respondents  and  to  charge  consultants 
and  marketeers  for  RFPs  as  a  cost  of 
doing  business; 

2)  establishing  the  Acquisition  Review 
Team  (ART)  as  the  sole  RFP  review 
at  ASD; 

3)  completing  the  initial  technical 
evaluation  before  determining 
whether  to  award  without  discussions; 

4)  severely  limiting  cost  analysis;  and, 

5)  deleting  the  requirement  for  Most 
Probable  Cost  (MPC)  estimates. 

The  general  recommendations  include: 

1)  development  of  an  extensive  training 
program  to  develop  a  thorough 
understanding  of  the  source  selection 
process ,  provide  practical  experience 
with  skills  required  in  source 
selection,  and  build  teamwork; 

2)  application  of  Total  Quality  Management 
(TQM)  tr,  the  source  selection  process  to 
ensure  that  all  decisions  and  activities 
provide  a  benefit  to  the  process;  and, 

3)  investigate  establishing  a  centralized 
source  selection  office  to  provide  core 
teams  of  experienced  personnel  to  conduct 
all  major  source  selections  in  order  to 
maintain  expertise  and  apply  it  to  every 
source  selection. 
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AM  ANALYSIS  OF  THE  SOURCE  SELECTION  PROCESS 


AT  AERONAUTICAL  SYSTEMS  DIVISION 

I .  Introduction 

When  the  Air  Force  identifies  a  requirement  for  the 
development  of  a  new  weapons  system,  the  contractor  to 
perform  the  development  is  determined  through  the  source 
selection  process,  a  disciplined  procedure  for  the 
solicitation  and  evaluation  of  offerors'  proposals 
designed  to  ensure  selection  of  ‘the  source  whose 
proposal  has  the  highest  degree  of  credibility  and  whose 
performance  can  be  expected  to  best  meet  the 
government’s  requirements  at  an  affordable  cost’  (2:3). 
In  fiscal  years  1986  and  1917,  the  Aeronautical  Systems 
Division  (ASD,  of  Air  Force  Systems  Command  (AFSC) 
conducted  17  formal  source  selections  for  new  systems 
ranging  from  Aircraft  Ground  Decoys,  canvas  or  wood 
mock-ups  of  fighter  aircraft,  to  the  President’s  new 
aircraft,  Air  Force  One,  and  the  next  generation 
fighter,  Advanced  Tactical  Fighter.  The  average  length 
of  time  required  to  complete  these  source  selections, 
from  release  of  the  solicitation  to  source  selection 
decision,  was  222  days,  almost  e’ ^ht  months  (11). 
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General  Issue 


Based  on  eleven  years  of  experience  in 
acquisition,  the  perception  of  the  researcher  is  that 
as  weapons  systems  and  the  rules  which  govern  their 
acquisition  have  growr  the  time  required 

to  complete  the  sour-  has  lengthened 

significantly.  Th‘  ha  source 

selection  process  d  ■C  4k  CVj  .  -  needed  contracts, 

resulting  in  increased  administrative  and  contract 
costs  to  the  Air  Force.  Perhaps  more  important,  this 
also  delays  acquisition  of  the  needed  operational 
capabilities  represented  by  the  new  weapons  systems. 

In  recognition  of  our  responsibility  to  provide 
the  taxpayer  with  the  maximum  value  for  each  tax  dollar 
spent,  AFSC  has  established  a  command  goal  of  120  days 
from  RFP  release  to  source  selection  decision.  The  Air 
Force  has  issued  a  new  regulation,  Air  Force  Regulation 
70-30,  Streamlined  Source  Selection  Procedures,  which 
suggests  using  limitations  to  proposal  length,  number 
of  evaluation  factors,  and  number  of  evaluators  to 
shorten  the  source  selection  cycle. 

During  the  first  quarter  of  fiscal  year  1988, 
fifteen  source  selections  were  completed  by  AFSC 
product  divisions  (10:2).  The  Medium  Launch  Vehicle 
II,  a  program  managed  by  AFSC 1 s  Space  Division,  has 
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been  cited  a3  an  example  of  the  successes  which  have 
been  achieved  with  the  new  streamlining  initiatives; 
the  source  selection  decision  was  rendered  only  114 
days  after  RFP  release  (10:2).  Other  programs  did  not 
fare  as  well.  For  example,  the  Mark  XV  source 
selection  required  221  days  from  RFP  release  to  source 
selection  decision  (1). 

specific  Issue 

In  order  to  consistently  complete  the  source 
selection  ir.  a  reasonable  amount  of  time,  changes  are 
required  in  source  selection  policies  or  in  the  source 
selection  process  itself. 

Investigative  Questions 

The  following  questions  were  addressed  to 
determine  how  the  source  selection  process  or  policies 
should  be  modified: 

1.  What  are  the  raa'or  factors  contributing  to 
the  length  of  the  process?  Can  these  factors  be 
control  led? 

2.  What  local  initiatives  have  been  implemented 
to  expedite  source  selections?  How  effective  have 
these  initiatives  been? 
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Scope  of  the  Research 


This  research  project  addresses  source  selection 
procedures  required  by  Air  Force  Regulations  70-15  and 
70-30  for  full-scale  engineering  development  and 
production  of  weapons  systems  over  *5  million  in 
estimated  cost.  Source  selection  procedures  for 
science  and  technology  (S&T)  programs  (basic  research, 
exploratory  development ,  and  advanced  technology 
development)  were  excluded  from  consideration;  due  to 
the  nature  of  S&T  efforts,  these  acquisitions  are 
generally  much  less  complex  than  weapons  system 
acquisitions  and,  therefore,  employ  simplified  source 
selection  procedures.  However,  recommended  changes 
resulting  from  the  research  may  in  some  cases  be 
applicable  to  S&T  procedures,  which  are  derived  from 
the  procedures  under  investigation. 

Only  those  situations  involving  technical 
competition  were  considered;  acquisitions  based  on 
price  competition  were  excluded  because  different 
procedures  are  utilized  when  low  price  is  the  sole 
criterion  for  selecting  the  successful  contractor. 
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I I .  Literature  Review 


Introduction 

Source  selection  requirements  and  policies  are  set 
forth  in  Subpart  15.6  of  the  Federal  Acquisition 
Regulation  (FAR);  Air  Force  Regulation  (AFR)  70-15, 
Formal  Source  Selections  for  Major  Acquisitions;  AFR 
70-30,  Streamlined  Source  Selection  Procedures;  and 
AFSC  and  product  division  supplements  to  these 
regulations?.  These  regulations  are  discussed  below  to 
provide  the  reader  with  an  understanding  of  the 
process  under  investigation.  This  discussion  is  not 
intended  to  address  overy  requirement  of  the 
acquisition  process,  but  will  provide  a  comprehensive 
delineation  of  source  selection  requirements.  The 
discussion  will  be  limited  to  these  regulations  because 
no  previous  research  on  the  source  selection  process 
was  identified. 

Discussion 

For  ease  of  presentation,  the  discussion  will  be 
divided  into  four  sections:  policy,  organization,  pre¬ 
evaluation  activities,  and  proposal  evaluation. 

Policy.  The  source  selection  procedures  described 
in  AFR  70-15  are  mandatory  for  Major  Defense 
Acquisition  Programs  and  Executive  Programs.  Major 
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Defense  Acquisition  Programs  are  those  programs 
estimated  to  exceed  *200  million  in  Research, 
Development,  Test,  and  Evaluation  (RDT&E)  and  *1 
billion  in  production  in  fiscal  year  (FY)  1980  dollars. 
In  FY88  dollars,  this  is  approximately  $300  million  in 
RDT&E  and  $1.6  billion  in  production  (2:3).  Other 
programs  which  do  net  meet  those  dollar  levels  may  be 
designated  as  Major  Defense  Acquisition  Programs  by  the 
Secretary  of  Defense,  Secretary  of  the  Air  Force,  or 
Assistant  Secretary  of  the  Air  Force  for  Acquisition. 
Executive  Programs  are  those  programs  selected  based  on 
'national  significance,  large  resource  commitment,  or 
management  complexity'  (2:3)  to  receive  direct 
oversight  by  the  Assistant  Secretary  of  the  Air  Force 
for  Acquisition.  These  programs  will  be  referred  to 
hereafter  as  'major  programs.' 

AFR  70-30  source  selection  procedures  are 
applicable  to  non-major  programs,  full-scale 
engineering  development  or  production  efforts  estimated 
to  cost  over  $5  million,  but  less  than  major  program 
levels . 
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The  source  selection  process  is  intended  to 
promote  competition;  minimize  the  complexity  of  the 
solicitation,  evaluation,  and  decision  process;  ensure 
an  impartial,  comprehensive  evaluation  of  proposals; 
and  ensure  selection  of  the  source  whose  proposal  is 
best  expected  to  fulfill  the  government's  requirements 
(2:3;  5:3;  7:15-20).  Consideration  must  be  given  to 
technical,  cost,  and  business  factors  (2:3;  5:3)  in 
determining  which  proposal  is  most  advantageous  to  the 
government . 

Organization .  Both  AFR  70-15  and  AFR  70-30 
provide  primary  and  alternate  organizations. 

Example  diagrams  of  the  two  organizations  are 
presented  below. 

Every  source  selection  is  directed  by  a  Source 
Selection  Authority  (SSA) ,  a  government  official 
designated  to  ensure  "proper  and  efficient  conduct  of 
the  entire  source  selection  process'  (5:4).  The  SSA 
has  sole  responsibi 1 i ty  for  making  the  source  selection 
decision.  Based  on  the  estimated  dollar  value  of  the 
acquisition,  importance  of  the  goods  or  services 
involved,  and  product  division  policies,  the  SSA  will 
be  designated  at  various  levels  ranging  from  the 
contracting  officer  to  the  Secretary  of  the  Air  Force. 
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SOURCE 
SELECTION 
AUTHORITY  (SSA) 


SOURCE  SELECTION  EVALUATION  TEAM  (SSET) 

CONTRACT 

TEAM 

TECHNICAL 

TEAM 

Figure  1.  Alternate  Organisation  for  AFR  70-15  Source 

Selection  (2:27) 

Primary  Organization  for  AFR  70-30  Source 
Selection  (5 : 19) 

The  Source  Select  on  Evaluation  Board  (SSEB) , 
shown  in  Figure  2,  is  an  ad  hoc  team  of  government 
specialists  responsible  for  evaluating  the  proposals 
against  the  minimum  requirements  of  the  solicitation. 
The  SSEB  is  divided  into  teams  which  perform  different 
functions.  The  cost  team  is  responsible  for  performing 
cost  and  price  analyses  and  developing  Most  Probable 
Cost  estimates  for  all  proposals.  The  contract 
def ini tization  team  serves  as  the  interface  between  the 
government  and  the  offerors,  and  is  responsible  for 
negotiating  contracts  with  all  acceptable  offerors. 

Each  technical  team  evaluates  a  different  aspect  of  the 
proposals,  (e.g.  engineering,  test,  logistics, 
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management)  excluding  contract  and  cost  matters.  These 
team  responsibilities  are  addressed  in  greater  detail 
in  the  sections  of  the  discussion  dealing  with  tho 
source  selection  activities. 


SOURCE 
SELECTION 
AUTHORITY  (SSA) 


ADVISORS 


SOURCE  SELECTION 
ADVISORY  COUNCIL  (SSAC) 

SOURCE  SELECTION  EVALUATION  BOARD  (SSEB) 


TEAMS  AND/OR  PANELS 


D 

E 

T 

T 

T 

T 

c 

F 

E 
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I 

T 

C 

C 

C 

C 

A 

A 

A 

A 

L 

L 

L 

L 

I 

0 

N 

* 

Figure  2.  Primary  Organization  for  AFR  70-15  Source 
Selection  (2:27) 

Alternate  Organization  for  AFR  70-30  Source 
Selection  (5 : 19) 
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The  Source  Selection  Advisory  Council  (SSAC) , 
Figure  2,  is  a  panel  of  senior  government  personnel 
appointed  by  the  SSA  to  provide  advice  on  conduct  of 
the  source  selection  and  to  prepare  a  comparative 
analysis  of  the  proposals  based  on  the  evaluation  by 
the  SSEB. 

In  the  organisation  shown  in  Figure  1,  the  SSEB 
and  SSAC  are  combined  into  the  Source  Selection 
Evaluation  Team  (SSET) .  The  SSET  is  composed  of  a 
contract  team,  which  performs  the  functions  of  the  cost 
and  contract  def ini tisation  teams  of  the  SSEB,  and  a 
single  technical  team  which  evaluates  ali  other  aspects 
of  the  proposals. 

AFR  70-15  prohibits  use  of  its  alternate 
organisation  (Figure  1)  when  the  Secretary  of  the  Air 
Force  acts  as  the  SSA  or  when  representatives  of  the 
Secretariat  participate  in  the  source  selection.  This 
prohibition  is  the  only  criterion  in  either  regulation 
concerning  selection  of  an  organisation  structure; 
neither  regulation  states  a  preference  for  either 
organisation. 
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AFSC  supplements  to  AFR  70-15  and  AFR  70-30 
require  a  Performance  Risk  Analysis  Group  (PRAG)  as  an 
independent  evaluation  group,  separate  from  the  SSEB, 
which  reports  directly  to  the  SSAC  (3:2)  or  as  a 
separate  evaluation  team  within  the  SSET  (6:  ).  The 
PRAG  13  composed  of  senior  government  personnel  with 
extensive  acquisition  experience.  The  PRAG  group  is 
responsible  for  evaluating  the  past  performance  of  each 
offeror  and  determining  the  performance  risk  associated 
with  each  proposal  (3:1;  6:4). 

Pre-evaluation  activities.  This  phase  of  the 
source  selection  includes  all  activities  from 
identification  of  the  requirement  through  receipt  of 
proposals . 

The  first  significant  activity  c  the  pre¬ 
evaluation  phase  is  the  Business  Strategy  Panel  (2:11; 
4:5)  or  Acquisition  Strategy  Panel  (3:1;  6:4).  These 
panels  are  intended  to  review  and  approve  the  proposed 
acquisition  strategy,  including  such  issues  as 
designation  of  the  SSA,  adequacy  of  the  technical 
requirements  documents,  source  selection  evaluation 
criteria,  quality  assurance  requirements,  business 
strategy,  logistics,  and  any  unusual  aspects  of  the 
acquisition  (5:5). 
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Prospective  sources  for  the  acquisition  must  be 
identified  through  market  surveys  which  may  range  from 
'contacts  with  knowledgeable  faderal  and  nonfederal 
exports  and  results  of  recent  market  tests  to  more 
formal  sources  such  as  ...announcements  ...in  the 
Commerce  Business  Daily*  (2:12). 

A  Source  Selection  Plan  (SSP)  describing  the 
program,  source  selection  organization,  pre-evaluation 
activities,  evaluation  procedures  and  criteria, 
acquisition  strategy,  and  schedule  for  completing  the 
source  selection  events  must  be  prepared  by  the  program 
office,  reviewed  by  organizations  determined  by  local 
policy,  and  approved  by  the  SSA. 

The  SSEB  or  SSET  must  develop  tho  criteria  by 
which  proposals  will  be  evaluated.  Throe  types  of 
evaluation  criteria  are  required:  cost  criterion, 
specific  criteria,  and  assessment  criteria. 

Cost  is  a  mandatory  criterion  to  be  evaluated  as 
an  area  for  every  source  selection.  Cost  and/or  price 
analysis  is  used  to  determine  the  reasonableness, 
realism,  and  completeness  of  the  proposed  price.  Cost 
is  not  rated;  results  of  the  cost/price  analysis  are 
presented  in  narrative  form  (2:14;  5:7). 
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Specific  criteria  are  derived  from  characteristics 
of  the  program.  Specific  criteria  deal  with  areas 
(e.g.  technical,  logistics,  management,  operational 
utility)  which  are  divided  into  items  representing  a 
greater  levol  of  detail.  Items  may  be  further  broken 
into  factors  and  subfactors  if  the  evaluation  area  is 
complex.  Only  characteristics  of  the  program  which  are 
significant  to  program  success  should  be  included  in 
the  specific  criteria  (2:14;  5:7). 


ITEM 


FACTOR 


ITEM 


FACTOR 


FACTOR 


SUBFACTOR 


FACTOR 


FACTOR 


SUBFACTOR 


Figure  3.  Graphic  depiction  of  structure  of  specific 
criteria. 
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Assessment  criteria  are  related  to  the  offeror's 


abilities  as  demonstrated  in  the  proposal  (2:14;  5:7). 
Assessment  criteria  typically  include  soundness  of 
approach,  understanding  the  problem,  compliance  with 
requirements,  and  special  technical  factors. 

Assessment  criteria  are  applied  against  the  specific 
criteria  in  an  evaluation  matrix  which  must  be 
developed  during  the  planning  phases  of  the 
acquisi tion . 


AREA:  TECHNICAL  DESIGN  AND  INTEGRATION 


SPECIFIC  CRITERIA 
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Figure  4.  Evaluation  Matrix  (2:32;  5:21). 
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Tho  relative  importance  of  the  evaluation  areas, 
specific  criteria,  and  assessment  criteria  must  be 
determined,  because  this  information  must  be  provided 
to  the  offerors  in  the  solicitation. 

In  addition  to  the  aspects  covered  by  the  criteria, 
general  considerations  such  as  past  performance, 
proposed  contractual  terms  and  conditions,  and  preaward 
survey  results  are  also  evaluated.  The  SSED  or  SSET 
must  determine  what  general  considerations  will  apply, 
because  these  considerations  must  be  provided  to  the 
offerors  in  the  solicitation. 

The  SSEB  or  SSET  must  devolop  evaluation  standards 
which  correspond  to  the  evaluation  criteria.  Tho 
standards  set  forth  the  minimum  acceptable  response  to 
a  requirement;  they  are  used  to  measure  whether  a 
proposal  meets,  exceeds,  or  fails  tc  neet  the 
requirements  (2:15;  5:8).  The  evaluation  standards  are 
not  included  in  the  solicitation  document. 

The  SSEB  or  SSET  must  prepare,  for  inclusion  in 
the  solicitation,  a  comprehensive  set  of  instructions 
for  proposal  preparation.  These  instructions  address 
content  and  format  of  the  information  to  be  provided  to 
the  government  for  evaluation. 
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The  final  pre-evaluation  activity  is  the 
preparation  and  issuance  of  the  solicitation  document, 
the  Request  for  Proposal  (RFP) .  The  contracting 
officer  is  responsible  for  preparing  the  RFP,  using 
information  and  documents  provided  by  the  program 
manager  and  other  team  members,  and  obtaining  the 
reviews  and  approvals  required  before  its  release. 

Proposal  evaluation  and  contract  award.  This 
phase  of  the  source  selection  process  begins  with  the 
receipt  of  proposals  from  the  offerors. 

If  desired,  the  SSEB  or  SSET  may  require  the 
offerors  to  conduct  oral  briefings  to  provide  the 
overviews  of  their  proposals.  Oral  presentations  must 
occur  before  the  evaluation  process  begins.  If  oral 
presentations  are  required,  evaluators  must  attend  all 
or  none  of  the  presentations  ‘to  eliminate  bias  and  to 
ensure  objectivity  during  the  evaluation  process* 

(2:  15)  . 

The  technical  evaluators  compare  each  proposal  to 
the  evaluation  standards  to  determine  the  adequacy  of 
each  proposal  and  assign  ratings  to  each.  The  team 
must  document  every  instance  in  which  a  proposal  fails 
to  meet  the  minimum  requirements  of  the  solicitation; 
these  documents  are  called  deficiency  reports.  The 
evaluators  must  also  document  any  aspects  of  the 
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proposal  which  require  clarification  because  the  data 
provided  is  contradictory  or  inadequate  to  evaluate; 
these  documents  are  called  clarification  requests.  The 
clarification  requests  are  immediately  sent  by  the 
contract  team  to  the  offeror  to  request  the  nooded 
clarification.  The  deficiency  reports  are  held  by  the 
contract  team  until  the  initial  technical  evaluation  is 
completed.  Disposition  of  these  reports  will  be 
addressed  below. 

The  products  of  this  evaluation  are  the  deficiency 

reports,  clarification  requests,  and  a  written  report 

which  must  address,  at  a  minimum: 

what  is  offered;  whether  it  meets  or 
fails  to  meet  the  standard;  any  strengths 
or  weaknesses;  what,  in  the  evaluator’s 
opinion,  may  be  done  to  remedy  a 
deficiency;  the  impact  of  any  deficiency; 
and  a  risk  assessment  of  the  offeror's 
ability  to  perform.  (2:15) 

The  evaluator  must  also  convert  the  assigned  ratings  to 
color  ratings  as  shown  in  Figure  5. 

The  cost  and/or  price  analysis  is  performed  by  the 
cost  team,  if  one  exists  in  the  organization,  or  the 
contract  team.  The  evaluation  typically  includes  both 
cost  and  price  analysis.  Cost  analysis  is  a  detailed 
examination  of  all  elements  of  cost  proposed  and 
requires  the  performance  of  an  audit  of  the  proposal  by 
the  Defense  Contract  Audit  Agency  and  field  pricing 
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report  by  the  contract  administration  office.  The 
findings  of  the  audit,  field  pricing  report,  and 
technical  evaluation  are  applied  to  the  cost  proposal 
to  determine  a  fair  and  reasonable  price  for  the 
proposed  effort.  Price  analysis  generally  compares  the 
proposed  price  of  the  system  to  prices  of  similar 
systems  to  determine  its  reasonableness.  On  major 
programs,  ‘an  additional  measurement  of  cost  or  price 
reasonableness  and  realism'  (2:17)  is  required.  This 
additional  check  requires  that  the  government  develop  a 
Most  Probable  Cost  estimate  which  is  based  on  the 
results  of  the  cost  analysis  of  the  proposed  costs  and 
represents  the  estimated  cost  of  ownership  of  the 
system  throughout  its  lifetime.  This  Most  Probable 
Cost  is  then  compared  to  the  proposed  cost  and  the 
previously  prepared  program  office  estimate. 

After  completion  of  the  initial  evaluation,  the 
contracting  officer  makes  a  competitive  range 
determination  for  approval  by  the  SSA  to  determine 
which  offerors  will  be  included  m  the  discussion  phase 
of  the  source  selection  process.  If  a  proposal  does 
not  have  a  reasonable  chance  of  being  selected  for 
award,  the  proposal  may  be  declared  to  be  outside  of 
the  competitive  range  and  excluded  from  further 
consideration  in  the  source  selection.  A  proposal  is 
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considered  to  be  outside  the  competitive  range  if  it 
does  not  address  essential  requirements  of  the 
solicitation,  is  so  deficient  that  further 
consideration  would  require  a  complete  revision  of  the 
proposal,  or  contains  major  deficiencies  which 
discussions  could  not  be  expected  to  cure  12:19;  5:10- 
11).  Affected  offerors  must  be  notified  immediately  of 
thoir  exclusion  from  the  competitive  range.  Following 
the  competitive  range  determination,  the  contract  team 
issues  the  deficiency  reports  to  the  appropriate 
offerors,  providing  an  opportunity  for  the  offerors  to 
revise  their  proposals  to  remedy  the  deficiencies. 

Discussions  are  conducted  by  the  contract 
def ini tization  team  of  the  SSEB  or  contract  team  of  the 
SSET.  Discussions  must  be  held  with  all  offerors 
determined  to  be  in  the  competitive  range.  During 
discussions,  offerors  must  be  given  an  opportunity  to 
correct  any  remaining  deficiencies  in  the  proposals  and 
to  resolve  any  uncertainties  and  suspected  mistakes. 

The  discussions  may  not  reveal  any  aspects  of  proposals 
submitted  by  other  offerors,  provide  notice  of  proposal 
weaknesses  caused  by  the  offeror's  lack  of  diligence  in 
preparing  the  proposal,  or  use  auction  techniques  to 
reach  an  expected  price  (7:15-21). 
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COLOR 

Blue 


Green 


Yellow 


Red 


RATING 

Exceptional 


Acceptable 


Marginal 


DEFINITION 

Excoeds  specified  performance 
or  capability  in  a  beneficial 
way  to  the  Air  Force;  and  has 
high  probability  of 
satisfying  the  requirement; 
and  has  no  significant 
weakness  . 

Meets  evaluation  standards; 
and  has  good  probability  of 
satisfying  the  requirement: 
and  any  weaknesses  can  bo 
readily  corrected. 

Fails  to  meet  evaluation 
standards;  and  has  low 
probability  of  satisfying  the 
requirement;  and  has 
significant  deficiencies  but 
correctable . 


Unacceptable  Fails  to  meet  a  minimum 

requirement;  and  deficiency 
requires  a  major  revision  to 
the  proposal  to  make  it 
correct . 


Figure  5.  Color  Codes  for  Proposal  Ratings  (2:16). 

When  discussions  with  all  offerors  are  complete, 
the  contracting  officer  affords  all  offerors  a  final 
opportunity  to  revise  the  proposals  through  the 
issuance  of  a  request  for  Best  and  Final  Offers 
(BAFOs) .  The  request  for  BAFO  includes  a  complete 
contract  which  the  offeror  must  sign  and  return  with 
the  BAFO. 
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In  the  BAFO,  an  offeror  may  modify  any  aspect  of 
the  proposal,  provided  that  adequate  rationale  for  any 
change  is  included.  After  receipt  of  the  BAF03 ,  the 
evaluators  must  update  the  evaluations  to  reflect  any 
changes  made  by  the  offerors. 

For  AFR  70-15  source  selections,  the  SSEB  must 
submit  the  results  of  the  evaluations  to  the  SSAC  in  an 
executive  summary  report  and  an  oral  briefing.  Based 
on  the  executive  summary  report  and  briefing,  the  SSAC 
prepares  an  analysis  report  which  compares  the 
proposals  and  presents  the  results  of  the  discussions 
and  BAFOs.  The  SSAC  is  also  responsible  for  preparing  a 
source  selection  briefing  based  on  the  comparative 
report  which  presents  to  the  SSA  all  the  information 
required  to  make  the  source  selection  decision  (2:20). 

When  the  procedures  of  AFR  70-30  are  used,  the 
SSET  prepares  a  Proposal  Analysis  Report  based  on  the 
evaluation  reports  prepared  by  the  evaluation  teams. 

The  SSET  also  presents  a  briefing  of  the  evaluation 
results  to  the  SSA  for  his  decision  (5:11). 
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The  SSA  decision  is  transmitted  to  the  cont 
officer  in  the  source  selection  decision  document 
signed  by  the  SSA.  The  document  sets  forth  the 
jocision,  rationale  for  the  decision,  and,  if  award  «3 
be  made  *.  other  than  the  low  offeror,  a 
determination  that  ‘the  technical  superiority  of  the 
higher  priced  proposal  warrants  the  additional  cost 
involved*  (2:21/22;  5:12;  ':10>. 

The  contracting  officer  executes  the  signed 
contract  submitted  with  the  winning  offeror's  BAFO, 
completes  any  final  contract  reviews  required,  provides 
the  congressional  and  other  notifications  required, 
notifies  the  unsuccessful  offerors,  and  effects 
distribution  of  the  fully  executed  contract.  The 
source  selection  is  complete. 
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Relationship  to  Research 


The  regulations  which  were  reviewed  in  this 
chapter  served  as  a  basis  for  the  list  of  activities 
and  sub-activities  which  was  developed  for  the 
interviews  used  in  gathering  the  data  for  this  study. 
These  regulations  also  served  as  a  basis  for 
analysing  the  source  selection  process  as  presented  by 
the  interview  subjects  and  their  suggested  changes  to 
the  process. 

The  methodology  used  in  developing  the  list  and 
performing  the  analyses  referred  to  above  are  addressed 
in  the  following  chapter. 
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III.  Methodology 


Overview 

This  chapter  describes  the  research  methodology 
used  to  answer  the  research  questions  posed  in  this 
study : 

1)  What  are  the  major  factors  contributing 
to  the  length  of  the  process?  Can  these  factors  bn 
controlled? 

2)  What  local  initiatives  have  been 
implemented  to  expedite  source  selections?  How 
effective  have  these  initiatives  been? 

The  discussion  includes  the  development  of  the 
interview  instrument,  subject  selection,  interview 
protocol,  and  data  analysis.  A  brief  summary  concludes 
the  chapter. 

Interview  Instrument  Development 

A  semi-structured  interview  was  determined  to  be 
the  appropriate  method  for  gathering  data  for  this 
field  experiment.  A  survey  was  inappropriate  because 
the  desired  information  was  not  quantitative  in  nature, 
and  could  only  be  obtained  through  verbal  explanation 
and  assessment  of  the  participants'  experiences.  A 
structured  interview  was  too  inflexible  for  the 


24 


circumstances.  The  semi-structured  interview  was 
determined  to  be  the  most  suitable  instrument  to  gather 
qualitative  information,  and  to  allow  follow-up 
questions  determined  by  the  responses  given. 

The  interview,  Appendix  A,  was  based  on  a  list  of 
source  selection  activities  which  was  derived  from  the 
regulations  discussed  in  Chapter  II  and  the 
researcher's  experience  with  source  selections  at 
Aeronautical  Systems  Division  (ASD) .  The  activities  on 
the  list  were: 

PRE-EVALUATION  ACTIVITIES 

Business  Strategy  Panel/Acquision  Strategy 
Panel 

Identification  of  sources 
Source  Selection  Plan 
Evaluation  criteria  development 
Evaluation  standards 
Instructions  to  offerors 
Request  for  Proposal  issuance 

PROPOSAL  EVALUATION  AND  CONTRACT  AWARD 
Receipt  of  proposals 
Quick  Look  briefing  or  memo 
Oral  brie^ngs  by  offerors 
Technical  evaluation 
Mid-Term  briefing 

Cost/price  analysis/Most  Probable  Cost 
estimate 

Competitive  range  determination 

Model  contracts 

Discussions 

Best  and  Final  Offers 

Technical  evaluation  update 

SSEB  Executive  Summary  Report 

SSEB  briefing  to  SSAC 

SSAC  Analysis  Report 

SSET  Proposal  Analysis  Report 

Final  SSA  briefing 

SSA  decision  document 

Contract  execution 
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Several  of  the  activities  were  further  divided  into  sub¬ 


activities,  A  complete  list  of  all  the  activities  and 
sub-activities  can  be  found  in  Appendix  A. 

A  list  of  basic  questions  was  developed  to  elicit 
the  information  needed  to  answer  the  investigative 
questions.  The  interview  questions  were  divided  into 
four  general  areas: 

1)  The  first  section  of  the  interview  was 
designed  to  identify  problem  areas  in  the  source 
selection  process.  The  repondents  were  required 
to  review  every  activity  on  the  list,  identify  the 
activities  with  which  they  had  no  experience,  rate  all 
other  activities  as  "not  a  problem*  or  as  "minor", 
"moderate",  or  "major*  problems.  For  each  activity 
identified  as  a  problem,  a  series  of  questions  was 
asked  concerning  the  nature  of  the  problem  experienced, 
the  cause  of  the  problem,  what  immediate  actions  were 
taken  to  deal  with  the  problem,  and  suggestions  for 
long-term  solutions  to  prevent  the  problem  in  the 
future . 

2)  The  second  section  of  the  interview 

was  designed  to  look  at  the  source  selection  activities 
from  a  cost/benefit  point  of  view;  that  is,  does  the 
benefit  derived  from  the  activity  justify  the 
investment  of  time  or  resources  needed  to  complete  it? 
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3)  The  next  portion  of  the  interview  addressed 
initiatives  which  had  been  implemented  to  avoid  or 
solve  recurring  problems  or  to  expedite  source 
selection  schedules. 

4)  The  .'final  portion  of  the  interview  offered  the 
subjects  the  opportunity  to  make  any  other  comments  on 
the  source  selection  process  or  the  interview. 

In  addition  to  this  information  on  the  source 
selection  process ,  limited  work-related  background 
information  was  gathered  on  the  interview  subjects. 

The  background  information  consisted  of: 

1)  job  title, 

2)  years  of  government  contracting 
experience , 

3)  other  non-buyer/PCO  acquisition- 
related  experience, 

4)  experience  at  other  AFSC  product 
divisions,  and, 

5)  extent  of  source  selection 
experience . 

This  data  was  collected  to  define  the  sample  from  which 
the  information  was  drawn. 

After  the  interview  questions  were  drafted,  thrp,* 
pretest  interviews  were  conducted,  using  the  same 
protocol  as  the  actual  interviews.  The  subjects  for 
the  pretest  interviews  were  ASD  contracting  personnel 
who  had  a  familiarity  with  source  selection  procedures, 
but  who  were  not  in  the  population  being  considered  by 
this  research;  these  subjects  had  conducted  source 
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selections  related  to  science  and  technology  (S&T) 
efforts.  The  pretest  interviews  were  conducted  to 
ensure  that  the  questions  were  easily  understood  and 
that  they  elicited  the  information  desired.  The 
interview  protocol  was  used  to  allow  the  researcher  to 
become  more  familiar  with  the  intended  protocol,  in 
order  to  be  more  fluent  and  at  ease  during  the  actual 
interviews.  No  interview  questions  were  revised  as  a 
result  of  the  pretest  interviews. 

Subject  Selection 

Because  this  was  a  purposive,  ncn-probability , 
judgmental  sampling,  subjects  were  chosen  for  their 
experience  with  the  topic  rather  than  by  random 
sampling.  Candidates  for  the  interview  were  ASD 
contracting  personnel  who  had  recent  experience  with 
source  selections  for  full-scale  engineering 
development  or  production  programs  over  *5  million. 

The  research  was  limited  to  ASD  because  of  time  and 
resource  constraints.  The  experience  requirement  and 
selection  of  ASD  as  the  research  location  determined 
the  size  of  the  population.  A  list  of  source  selections 
which  had  been  completed  within  the  last  18  months  was 
obtained  from  ASD/PMPS ,  the  ASD  source  selection 
office.  Seventeen  source  selections  and  18  points  of 
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contact  were  included.  The  individuals  listed  for  each 


source  selection  were  contacted  to  schedule  interviews. 
In  several  cases,  the  original  participants  in  the 
source  selections  had  been  re-assigned  to  different  ASD 
organizations,  and  the  listed  individual  had  begun  to 
work  on  the  program  after  the  source  selection  had  been 
completed.  The  names  of  the  original  participants  were 
obtained,  and  a  total  of  19  individuals  were  identified 
as  the  population.  Of  the  19  possible  subjects,  one 
individual  was  an  Air  Force  officer  who  had  been  re¬ 
assigned  from  ASD,  and  was  thus  no  longer  available  to 
be  interviewed.  One  individual  refused  to  participate. 
Two  individuals  agreed  to  be  interviewed,  but  did  not 
appear  for  the  interviews.  Three  individuals  were 
unavailable  due  to  workload  or  temporary  duty 
considerations.  A  total  of  12  individuals  participated 
in  the  interviews. 

Protocol 

At  least  one  day  before  the  interviews,  the 
subjects  were  provided  copies  of  the  interview  package 
which  is  included  as  Appendix  A.  It  was  requested  that 
they  complete  the  section  on  work  experience  before  the 
interview  started.  The  lists  of  source  selection 
activities  and  questions  were  provided  to  allow  the 
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subjects  to  become  familiar  with  the  areas  to  be 
addressed.  The  subjects  were  also  asked  to  identify 
the  activities  with  which  they  had  no  experience. 

These  actions  were  intended  to  minimize  the  time 
required  to  conduct  the  interviews. 

The  interviews  were  conducted  face  to  face  at  ASD. 
Responses  were  recorded  in  writing  during  the 
interviews;  the  instruments  used  by  the  researcher  had 
been  designed  to  facilitate  note-taking,  and  the  pace 
of  the  interviews  was  slow  enough  to  allow  adequate 
written  documentation. 

At  the  start  of  the  interview,  the  participants 
were  told  that  their  names  would  be  listed  as  interview 
subjects,  but  that  their  responses  would  be  anonymous; 
no  program  specific  information  would  be  included  in 
Chapter  IV,  and  no  responses  would  be  tied  to  any  names 
of  the  interview  subjects.  This  guarantee  of  anonymity 
was  believed  by  the  researcher  to  be  necessary  to 
ensure  that  honest,  open  responses  were  provided. 
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The  subjects  were  generally  very  cooperative  and 
appeared  to  be  interested  in  providing  extensive  and 
reliable  information.  It  appeared  to  the  researcher 
that  the  participants  felt  that  they  had  valuable 
information  to  contribute,  and  that  their  participation 
could  provide  the  basis  for  meaningful  change  in  the 
source  selection  process. 

During  the  interview,  the  researcher  frequently 
used  follow-up  questions  to  elicit  further  information 
about  problem  areas.  The  individual  follow-up 
questions  were  based  on  the  responses  given  by  the 
subjects  and  the  types  of  information  desired. 

For  ease  of  conducting  the  interview,  the  list  of 
activities  was  divided  into  two  sections:  1)  pre¬ 
evaluation  activities,  and  2}  proposal  evaluation  and 
contract  award.  Each  section  of  the  list  was  handled 
separately . 

Analysis 

Statistical  analysis  of  the  responses  was  limited 
to  frequency  distributions  of  the  'major,  moderate, 
minor,  not  a  problem,  no  experience'  responses.  Every 
source  selection  activity  identified  as  a  problem  of 
any  magnitude  was  addressed  in  narratives  which 
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summarise  and  compare  the  interview  responses.  Any 
suggested  long-term  solutions  to  these  problems  were 
analysed,  both  for  compatibility  with  requirements  of 
law  and  practicality  of  implementation. 

The  results  of  the  analysis  of  the  interview 
responses  were  used  to  compile  a  list  of 
recommendations  for  changes  in  regulation,  policy,  or 
procedures  and  suggestions  for  further  research. 

Chapter  Summary 

This  chapter  described  the  methodology  used  in 
conduct  of  this  research  project  from  formulation  of 
the  interview  questions  through  data  analysis.  The 
results  of  this  methodology  will  be  presented  in 
Chapter  IV. 
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IV.  Findings  and  Analysis 


Overview 

This  chapter  presents  the  research  findings 
resulting  from  the  previously  described  methodology. 
Following  a  description  of  the  sample  from  which  the 
information  was  obtained,  the  results  of  the  interviews 
will  be  presented  and  analyzed.  The  interview  was 
divided  into  four  areas: 

1)  source  selection  problem  areas  and 
suggested  solutions; 

2)  cost/benefit  of  source  selection 
activities ; 

3)  initiatives  to  improve  source  selection; 
and , 

4)  general  comments  on  the  process. 

The  discussion  of  the  interview  findings  will  be 
divided  in  the  same  manner.  A  brief  summary  will 
complete  the  chapter. 


Description  of  the  Sample 

The  sample  for  this  research  project  consisted  of 
12  individuals.  While  this  is  a  small  sample,  the 
total  population  consisted  of  only  19  people.  The 
individuals  in  the  sample  displayed  considerable  * 
knowledge  on  the  source  selection  process,  were  very 
forthcoming  in  their  responses,  and  provided  an 
adequate  sample  for  a  pilot  research  study  such  as 
this . 
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The  interview  subjects  had  worked  as  government 
contract  negotiators  and/or  contracting  officers  for  an 
average  of  over  12.5  years.  The  range  of  experience 
was  from  3.5  to  27  years.  The  distribution  of 
experience  is  shown  in  Table  1.  One  individual  cited 
1.5  years  of  contracting  experience  with  Air  Force 
Logistics  Command;  the  other  individuals  had  worked 
only  at  ASD.  All  the  interview  subjects  were 
civilians;  one  Air  Force  officer  was  a  member  of  the 
population  but  was  no  longer  assigned  at  ASD.  No 
members  of  the  sample  had  non-buyer/PCO  acquisition- 
related  experience.  More  than  half  the  sample  had 
participated  in  only  one  major  source  selection;  the 
average  for  the  sample  was  1.9,  with  a  maximum  of  four. 
This  information  is  shown  in  Table  2.  Many  individuals 
in  the  sample  had  also  participated  in  source 
selections  which  are  outside  the  scope  of  this  study 
(under  *5  million  or  science  and  technology) . 

All  individuals  in  the  sample  had  an  adequate 
amount  of  general  contracting  experience  to  be 
considered  credible.  While  source  selection  experience 
was  rather  limited,  this  was  to  be  expected;  many 
contracting  offices  at  ASD  do  very  limited  numbers  of 
major  source  selections,  if  any.  The  greatest  number 
of  source  selections  are  conducted  in  organizations 
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such  as  Aeronautical  Equipment  (ASD/AE)  or  Training 
Systems  (ASD/YW) ,  which  manage  a  number  of  programs. 
Offices  which  manage  a  single  large  program,  such  as 
the  F-15  (ASD/VF)  or  F-16  (ASD/YP)  ,  conduct  most  of 
their  business  through  sole  source  contracts  with  the 
airframe  manufacturers.  Therefore,  the  possibility 
of  obtaining  experience  in  source  selection  is  related 
to  the  organisations  to  which  a  contract  negotiator  is 
assigned  and  the  frequency  of  job  rotation.  An 
additional  factor  in  source  selection  experience  is  the 
limited  number  of  major  source  selections  conducted  by 
the  entire  product  division.  Only  seventeen  major 
source  selections  were  completed  at  ASD  between 
February  1988  and  June  1989. 

Table  1.  Contracting  experience  of  sample  members. 

Experience  No.  of  Respondents 

(in  years) 

1-5  1 


6 

-  10 

5 

1  1 

-  15 

4 

16 

-  20 

0 

21 

-  25 

1 

26 

-  30 

1 
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Table  2.  Source  selection  experience  of  sample. 


Ho.  of  source  selections  Ho.  of  respondents 


1 

7 

2 

0 

3 

4 

4 

l 

Problem  Areas  and 

Suggested  Solutions 

For  this  portion  of  the  interview,  subjects 
identified  source  selection  activities  with  which  they 
had  no  experience,  and  rated  the  ether  source  selection 
activities  as  ‘not  a  problem'  or  as  'minor', 

'moderate*,  or  'major*  problems.  Two  of  the 
respondents  also  insisted  on  rating  several  items  as 
'moderate  to  major'  and  'minor  to  moderate,*  although 
these  were  not  among  the  desired  responses.  The 
summary  of  the  responses  to  this  portion  of  the 
interview  is  detailed  in  Appendix  B.  Tor  each 
activity  rated  as  a  problem,  the  problems  experienced, 
causes  of  those  problems,  how  the  problems  were 
addressed,  and  how  they  might  be  avoided  in  the  future 
were  discussed. 

Only  five  individuals  cited  activities  with  which 
they  had  no  experience.  This  information  was 
requested  because  not  all  the  activities  listed  are 
included  in  all  source  selections;  the  researcher  did 


not  want  to  show  an  activity  as  'not  a  problem*  if  the 
lack  of  problems  could  have  been  due  to  a  lack  of  use  of 
the  activity.  However,  a  total  of  only  16  activities 
were  cited.  The  maximum  number  of  'no  experience’ 
responses  for  any  activity  was  two.  These  responses 
are  shown  in  Table  3. 


37 


A  total  of  94  problems  were  identified  by  the 
respondents.  The  average  number  of  problems  per 
interview  was  7.8,  with  a  range  from  1  to  16.  The  most 
fiequently  identified  problem  activity  (including  its 
sub-activities)  was  'technical  evaluation,'  named  19 
times.  Only  five  activities  were  rated  as  'not  a 
problem*  by  all  respondents  having  experience  with 
these  activities: 

1)  oral  briefings  by  offerors, 

2)  competitive  range  determination, 

3)  SSEB  Executive  Summary  Report, 

4)  SSEB  briefing  to  SSAC,  and, 

5)  SSET  Proposal  Analysis  Report. 

Each  activity  cited  as  a  problem  will  be  addressed 
separately.  For  the  convenience  of  the  reader,  the 
number  of  problem  responses  is  shown  after  each 
activity  name.  The  discussion  for  each  problem 
activity  is  divided  into  two  sections.  The  first 
section  provides  the  information  derived  from  the 
interviews.  The  second  section  contains  the 
researcher's  analysis  of  that  information. 

Business  Strategy  Panel  (BSP) /Acqulsi tion  Strategy 
Panel  (ASP) .  (Moderate  -  1;  Minor  -  3) 

Responses .  The  moderate  problem  concerned 
guidance  given  by  the  ASP.  The  respondent  felt  that 
the  guidance  received  is  not  always  logical  and  may 
result  m  increased  costs  for  the  program.  Examples  of 
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such  guidance  included  changing  the  contract  type  for  a 
development  effort  to  firm  fixed  price  and,  in  another 
instance,  changing  the  basis  for  award  to  place  more 
emphasis  on  technical  aspects  when  the  acquisition  team 
had  Judged  cost/price  to  be  the  most  important  factor 
in  selecting  the  successful  offeror.  The  cause  cited 
by  the  respondent  was  politics  overcoming  good  business 
decisions.  At  the  time,  the  participants  attempted  to 
influence  the  outcome  of  the  ASP;  after  failing  in 
that,  the  only  means  of  dealing  with  ASP  guidance  which 
was  opposed  to  the  team's  judgment  was  to  accept  and 
implement  the  guidance  given.  The  respondent's 
proposed  solution  was  to  ensure  that  good  business  (as 
opposed  to  political)  decisions  are  made  up  front.  Mo 
specific  means  of  implementing  this  solution  was 
provided . 

A  minor  problem  cited  for  the  BSP/ASP  activity 
concerned  the  inability  to  obtain  a  waiver  to  the 
requirement  to  conduct  a  BSP  for  a  program  which  was 
considered  routine  and  had  no  unusual  aspects.  The 
cause  of  this  failure  was  unknown  to  the  respondent.  A 
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BS?  was  conducted,  which  resulted  in  a  schedule  slip. 
The  proposed  long-term  solution  included  a  more  liberal 
waiver  policy  for  acquisitions  which  are  routine  in 
nature,  and  better  planning  by  the  acquisition  team  to 
accomodate  unforeseen  occurrences  such  as  the  inability 
to  obtain  waivers,  so  that  schedule  impacts  could  be 
mitigated . 

Another  minor  problem  involved  conflicting 
interpretations  of  the  recommendations  made  by  the 
BSP.  In  this  case,  team  members  held  a  meeting 
during  which  a  consensus  was  reached.  This  problem 
has  since  been  solved  by  implementation  of  an 
administrative  requirement  which  provides  an  uninvolved 
party  to  attend  the  panel  and  record  the  minutes. 

The  third  minor  problem  concerned  the  lack  of 
adequate  planning  by  the  acquisition  team;  the  team  did 
not  begin  preparation  far  enough  in  advance  of  the  ASP. 
The  cause  of  this  problem  involved  workload 
considerations;  team  members  had  other  workload  for 
which  they  were  responsible,  and  ASP  preparation  was 
not  made  a  priority  over  other  workload.  The  problem 
was  solved  by  having  the  SPO  director  assign  a  priority 
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to  ASP  preparation.  The  respondent  suggested  earlier 
identification  of  the  potential  problem  caused  by 
delaying  ASP  preparation,  and  earlier  assignment  of  a 
priority  to  this  preparation  activity  to  prevent  this 
problem  in  the  future. 

Analysis .  Two  of  the  four  problems  related 
to  the  BSP/ASP  involved  the  unwillingness  of 
authorities  outside  of  a  program  office  to  accept  the 
judgment  of  those  most  knowledgeable  about  the 
acquisition.  While  oversight  by  uninvolved  experts 
can  provide  a  useful  balance  to  the  parochial  views  of 
the  program  team,  these  strategy  reviews  sometimes 
serve  ’political’  interests  at  the  expense  of  good 
business  judgment.  This  would  appear  to  be  an  arena 
where  the  decentralization  of  decision-making 
could  provide  a  benefit  to  the  acquisition  process. 
Because  numerous  management  reviews  are  required  for 
major  source  selections,  it  would  appear  that  deletion 
of  the  BSP/ASP  requirement  for  these  acquisitions  would 
not  jeopardize  management  oversight  responsibi 1 i ties . 
Changes  to  AFR  70-15,  AFR  70-30,  and  the  AFSC 
regulation  on  Acquisition  Strategy  Panels  would  be 
needed  to  delete  the  BSP/ASP  requirement  for  source 
selections . 
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Identification  of  Sources.  (Moderate  -  1; 


Minor  -  4) 

Responses .  A11  respondents  cited  the  same 

issue  related  to  identification  of  sources.  The 
regulatory  requirements  for  this  activity  include  a 
market  survey,  which,  because  its  formal  definition  is 
nebulous,  is  generally  conducted  through  a  formal 
advertisement  in  the  Commerce  Business  Daily,  and  a 
synopsis  required  by  the  Competition  in  Contracting  Act 
(CICA) ,  which  must  be  accomplished  by  advertising  in 
the  Commerce  Business  Daily.  The  market  survey  is 
meant  to  discover  qualified  sources  for  the 
acquisition;  only  those  sources  who  appear  to  have  the 
requisite  capabilities  are  placed  on  the  source  list. 
The  CICA  synopsis  requirements  allow  no  screening  of 
potential  sources;  all  respondents  are  placed  on  the 
source  list  and  receive  copies  of  the  solicitation 
documents.  In  effect,  two  virtually  identical  synopses 
are  required,  which  result  in  numerous  unqualified 
sources  on  the  source  list.  This  problem  was  caused  by 
the  unclear  definition  of  a  market  survey  and  the  lack 
of  ability  to  screen  respondents  to  the  CICA  synopsis. 
The  usual  method  of  dealing  with  this  problem  has  been 
to  publish  two  synopses  and  send  RFPs  to  all 
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Suggestions  for  long-torm  solutions 


respondents . 
included : 

1)  combining  the  market  survey  and  synopsis 
requirements  to  allcv*  one  announcement#; 

2)  changing  the  FAR  requirements  for  the 
CICA  synopsis  to  allow  screening  of  the 
capabilities  of  the  respondents  to 
limit  the  source  list  to  qualified 
sources;  and, 

3)  changing  the  FAR  requirements  for  the 
CICA  synopsis  to  require  respondents  to 
identify  themselves  as  manufacturers  or 
marketeers/consultants  and  to  require 
non-manufacturing  concerns  to  pay  for 
solicitation  documents  as  a  cost  of 
doing  business. 

Analysis .  All  throe  suggested  solutions 
appear  to  be  reasonable  approaches  to  this  problem. 

The  most  conservative  suggestion,  combining  the 
announcements,  would  eliminate  one  step  from  the 
acquisition  process  and  result  in  a  minor  reduction  of 
the  time  involved.  The  ability  to  screen  respondents 
and/or  charge  non-manufacturing  concerns  for  RFPs  would 
provide  a  greater  benefit  to  the  acquisition  system; 
the  administrative  effort  required  for  reproduction  and 
mailing  dozens  of  solicitation  documents,  when  only  two 
or  three  proposals  are  exp  cted,  is  a  burden  on  the 
acquisition  system  which  involves  no  return  o" 
investment.  It  could  be  argued  that  allowing  the 
elimination  of  prospective  sources  could  unfairly 
restrict  access  to  the  contracting  system;  however, 
criteria  for  screening  could  be  developed  to  minimize 
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the  possibilities  for  abuse.  Implementation  of  any  of 
these  suggestions  would  require  a  change  to  the  FAR; 
changes  to  the  CICA  synopsis  might  require  changes  to 
public  law,  depending  on  the  detail  included  in  CICA 
addressing  the  synopsis  requirements. 

Source  Selection  Flan  (SSP) .  (Major  -  1; 

Moderate  -  1;  Minor  -  1) 

Responses .  The  major  problem  related  to  the 
SSP  concerned  the  time  required  for  preparation  of  the 
plan  due  to  the  number  of  revisions  required.  This 
problem  was  caused  by  the  lack  of  knowledge  of  the 
planning  process  and  the  required  content  and  format  of 
the  SSP  on  the  part  of  personnel  charged  with  preparing 
the  plan.  The  problem  was  corrected  through  meetings 
with  the  program  office  to  work  out  the  problems.  The 
suggested  long-term  solution  was  education;  the 
respondent  suggested  that  a  short  course  be  developed, 
based  on  ASD  source  selection  procedures,  to 
familiarize  participants  with  SSP  requirements. 

The  moderate  problem  ’ealt  with  the  inclusion  of 
too  many  evaluation  factors  in  the  SSP.  The  evaluators 
responsible  for  these  inputs  to  the  plan  wanted  to 
evaluate  too  many  aspects  of  the  proposals;  they  had  to 
be  pursuaded  that  the  streamlining  initiatives  required 
a  reduction  in  the  scope  of  the  technical  evaluation. 
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A  meeting  between  the  appropriate  directors  was 
required  to  accomplish  this  end.  The  long-term 
solution  for  this  problem  involved  enforcement  of 
existing  regulatory  guidance  that  limits  the  number  of 
evaluation  factors. 

The  minor  problem  was  very  similar  to  the  major 
problem  discussed  above.  In  addition  to  the  previously 
cited  cause,  this  respondent  also  cited  lack  of  early 
involvement  of  contracting  personnel  as  a  cause  of  the 
problem.  In  addition  to  suggesting  education  of  the 
plan  preparers  as  to  format  and  contents  of  the  plan, 
this  respondent  suggested  early  involvement  of 
contracting  personnel  in  the  planning  process  as  a 
means  of  avoiding  problems  with  SSP  preparation. 

Analysis .  Earlier  involvement  of  contracting 
personnel  in  the  planning  process  for  source  selections 
is  needed  to  ensure  that  all  contracting  issues  are 
adequately  considered.  The  failure  of  program 
management  personnel  to  seek  this  involvement  can  be 
caused  by  conflict  between  the  program  management  and 
contracting  functions  which  results  from  the  diff-erent 
roles  which  are  served  by  the  functions.  This  conflict 
must  be  overcome  so  that  the  teamwork  which  is 
necessary  for  efficient  conduct  of  source  selections 
can  be  achieved. 
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The  lack  of  knowledge  of  the  source  selection 
process  cited  by  these  respondents  reflects  a  common 
theme  of  the  interviews.  While  the  SSP  in  itself  does 
not  represent  a  significant  problem  in  the  source 
selection  process,  the  ignorance  which  surrounds  the 
planning  to  be  reflected  in  the  SSP  is  a  significant 
factor;  this  ignorance  will  be  cited  again  and  again 
throughout  this  study.  The  formal  training  suggested 
to  eliminate  problems  with  the  SSP  would  require  an  Air 
Force  commitment  of  additional  resources  for  developing 
and  conducting  the  training  course,  but  is  desireable 
to  achieve  improvements  in  the  performance  of  source 
selection  participants. 

Evaluation  criteria  development.  (Major  -  3; 


Moderate  -  3) 

Responses .  All  respondents  cited  similar 
problems  with  evaluation  criteria  development.  The 
cited  problems  included  criteria  that  wore: 

1)  inaccurate; 

2)  not  limited  to  items  which  were 
important  to  the  evaluation; 

3)  appeared  to  favor  a  particular  approach 
or  contractor; 

4)  too  specific; 

5)  inappropriate; 

6)  not  well  thought-out; 

7)  not  well  written; 

8)  required  numerous  re-writes;  and, 

9)  did  not  reflect  the  proper  priorities 
of  the  technical  evaluation  or 
correlate  with  the  evaluation  standards 
and  instructions  to  offerors. 
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The  most  frequently  cited  cause  for  these  problems  was 
a  general  lack  of  understanding  on  the  part  of  program 
managers  and  engineers  of  the  source  selection  process 
and  how  evaluation  criteria  fit  into  that  process  (and 
relate  to  other  documents  such  as  the  evaluation 
standards  and  instructions  to  offerors).  Another  cause 
cited  by  several  respondents  was  personnel  turnover; 
the  lack  of  team  continuity  from  year  to  year  results 
in  the  loss  of  learning  and  need  to  train  new 
participants  for  every  source  selection.  An  element  of 
this  cause  was  the  emphasis  placed  on  source  selection 
experience  in  the  promotion  process;  several 
respondents  felt  that  this  emphasis  encouraged 
personnel  to  participate  in  one  source  selection  as  a 
square-filling  exercise,  which  prevents  the  development 
of  expertise  in  the  process.  The  problems  were 
resolved  through  numerous  reviews  and  corrections  of 
the  criteria  and  through  team  meetings  to  thrash  out 
the  problems.  Suggested  solutions  centered  on 
education.  Three  of  the  six  respondents  suggested  that 
ASD/PMPS ,  the  Source  Selection  Division,  develop  a 
short  course  on  source  selection  to  be  conducted  very 
early  in  the  planning  stages.  The  course  should  focus 
on  planning  for  source  selections  and  provide 
instruction  in  preparation  of  the  required  documents 
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and  how  they  relate  to  each  other.  One  respondent 
suggested  that  completion  of  the  Statement  of  Work  and 
specification  be  required  before  development  of 
evaluation  criteria  could  begin.  Another  suggestion 
proposed  that  the  evaluation  criteria  be  approved  at 
the  ASP  to  require  early  development  and  multi¬ 
disciplinary  review  of  the  criteria. 

Analysis .  Half  of  the  respondents  cited 
significant  problems  with  evaluation  criteria 
development.  These  problems  flow  throughout  the  process 
because  many  other  documents  or  activities  build  on  or 
relate  to  the  criteria.  Elimination  of  the  problems 
with  this  activity  could  provide  a  significant  benefit 
to  the  entire  process. 

The  problems  cited  for  evaluation  criteria 
development  re-inforce  the  need  for  formal  training  to 
provide  an  understanding  of  the  source  selection 
process.  If  the  participants  do  not  understand  the 
process,  it  is  unlikely  that  they  will  see  the 
relationships  between  the  various  documents  and 
activities.  An  understanding  of  the  process  should 
eliminate  the  failure  to  tie  the  evaluation  criteria  to 
the  requirements  documents  of  the  RFP ,  the  evaluation 
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standards,  and  the  instructions  to  offerors.  The 
participants  would  be  better  able  to  write  objective, 
relevant,  and  meaningful  criteria. 

The  issue  of  personnel  turnover  and  resultant  lack 
of  source  selection  expertise  were  not  addressed  by  the 
respondents  in  their  long-term  solutions  for  dealing 
with  problems  in  developing  evaluation  criteria.  This 
situation  will  be  addressed  in  the  analysis  of 
technical  evaluation  problems  and  in  the  fourth  section 
of  this  chapter,  which  presents  the  general  comments  of 
the  respondents. 

Evaluation  standards.  (Major  -  3; 

Moderate  -  1) 


Responses .  Problems  cited  for  evaluation 
standards  were  very  similar  to  those  cited  for 
evaluation  criteria.  The  cited  problems  included 


standards  that  were: 

1 )  unfair; 

2)  not  definitive; 

3)  not  specific; 

4)  inaccurate; 

5)  did  not  reflect  the  RFP  requirements; 
and , 

6)  did  not  correlate  with  the  evaluation 
criteria  and  instructions  to  offeror. 


Cited  causes  for  these  problems  included: 

1)  a  lack  of  understanding  of  the  source 
selection  process; 

2)  lack  of  familiarity  with  the  technical 
requirements  of  the  RFP; 
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3)  ignorance  of  the  relationship  between 

various  parts  of  the  RFP  (that  standards 
flow  from  technical  requirements  and  <& 

evaluation  criteria) ; 

4)  preparation  of  related  RFP  documents 
by  different  personnel  and  as  separate 
enti ties ; 

5)  lack  of  training;  and, 

6}  time  constraints. 

Two  respondents  stated  that  t>e  problems  were  resolved 
through  team  meetings  to  refine  the  standards.  One 
respondent  was  unable  to  influence  the  situation.  In 
the  final  instance,  the  problems  with  the  standards 
were  not  discovered  until  the  technical  evauation  was 
in  process  and  the  bad  standards  made  the  evaluation 
impossible.  The  RFP  and  evaluation  standards  had  to  be 
modified,  and  the  technical  evaluation  re-accomplished. 
Long-term  solutions  centered  around  formal  training  to 
educate  team  members  about  the  purpose  and  development 
of  evaluation  standards.  Another  suggestion  concerned 
ensuring  that  team  members  responsible  for  preparing 
the  standards  are  knowledgeable  of  the  technical 
requirements  in  the  RFP.  It  was  also  recommended  that 
the  Statement  of  Work  and  specifications  be  completed 
before  development  of  the  evaluation  standards. 

Analysis .  One-third  of  the  respondents  had 
experienced  significant  problems  related  to  evaluation 
standards.  Again,  many  of  the  problems  resulted  from  a 
lack  of  understanding  of  the  source  selection  process 
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and  the  relationships  between  the  various  documents  and 
activities.  These  problems  could  be  mitigated  through 
formal  education. 

An  additional  problem  concerned  the  lack  of 
familiarity  of  technical  team  members  with  the 
technical  requirements  of  the  RFP ,  resulting  in  the 
development  of  standards  which  did  not  reflect  the  RFP 
requirements.  This  problem  would  seem  to  indicate  a 
lack  of  diligence  on  the  part  of  the  technical  team 
chief,  program  manager,  and/or  SSEB  chairman.  These 
individuals  should,  as  part  of  their  reviews  of  the 
source  selection  documents,  ensure  the  accuracy  of  all 
standards  by  checking  the  sources  of  those  standards  in 
the  technical  requirements  documents.  This  problem 
could  have  been  influenced  by  time  constraints,  which 
were  cited  by  respondents;  quality  may  have  been 
sacrificed  to  schedule. 

Instructions  to  offerors  (ITO).  (Major  -  2; 
Moderate  -  2) 

Responses .  Both  respondents  who  cited  major 
problems  with  the  ITO  focused  on  the  scope  of 
information  required  by  the  ITO.  In  both  cases,  the 
instructions  required  the  offerors  to  submit 
significant  amounts  of  data  which  would  not  or  could 
not  be  evaluated.  In  one  case,  the  ITO  was  100  pages 
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in  length,  single-spaced,  which  is  the  same  number  of 
pages  the  offeror  was  allowed  for  the  technical 
proposal.  An  additional  problem  related  to  the  failure 
of  the  ITO  to  coincide  with  the  evaluation  criteria  and 
standards.  Those  who  cited  moderate  problems  with  the 
ITO  found  that  extensive  changes  were  required  to  the 
instructions;  in  one  case,  the  drafters  attempted  to 
mandate  contractual  requirements  through  the  ITO. 

Causes  cited  by  the  respondents  included: 

1)  a  lack  of  understanding  of  the  source 
selection  process; 

2)  lack  of  knowledge  of  the  ‘canned*  ITO, 
which  led  to  inclusion  of  unneccessary 
data  and  deletion  of  needed  information; 

3)  lack  of  integrated  review  of  the  ITO; 

4)  lack  of  knowledge  concerning  what 
information  is  needed  to  demonstrate 
compliance  with  standards; 

5)  lack  of  training;  and, 

6)  time  constraints. 

Team  review  of  the  ITO  and  a  meeting  between  the  SSEB 
deputy  chairman  and  the  area  chiefs  were  used  to  deal 
with  these  problems.  Suggestions  for  long-term 
solutions  included  early  meetings  between  functionals  ^ 

and  area  chiefs  to  foster  a  true  understanding  of  the 
contents  and  purpose  of  the  ITO,  and  formal  training. 

One  individual  suggested  the  formation  of  small  inter¬ 
disciplinary  teams  to  prepare  the  ITO  as  an  integrated 
document,  rather  than  piece-mealing  modified  boiler¬ 
plate  ITO  segments  together. 
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Analysis .  One-third  of  the  respondents  also 
had  significant  problems  with  the  ITO.  These  problems 
were  often  related  to  the  problems  experienced  with 
evaluation  criteria  and  standards,  and  had  the  common 
cause  of  lack  of  understanding  of  the  source  selection 
process.  The  problems  with  the  ITO  involve  all 
functional  disciplines  of  the  source  selection  team; 
the  ITO  consolidates  the  data  needs  of  every  functional 
area.  In  addition  to  formal  education,  preparation  of 
an  integrated  ITO  by  a  small  inter-disciplinary  team 
drawn  from  the  SSEB  or  SSET  would  appear  to  be  a  viable 
approach  to  minimizing  the  problems  encountered. 

Purchase  Request  (PR)  package  development. 

(Major  -  4;  Moderate  -  2;  Minor  -  2) 

Responses .  Three  of  the  four  respondents 
who  characterized  PR  package  development  as  a  major 
problem  stated  that  the  requirements  packages  delivered 
by  the  program  office  to  contracting  were  deficient. 

The  packages  were  missing  documents  or  were  disjointed, 
with  no  flow  between  related  parts  such  as  the 
Statement  of  Work,  evaluation  criteria,  instructions  to 
offerors,  and  basis  for  award.  The  packages  required 
major  corrections  before  they  were  suitable  for  use  by 
the  contracting  offices.  Causes  cited  included: 
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1)  lack  of  knowledge  of  the  required 
contents  of  a  PR  package; 

2)  lack  of  a  definitive  operating 
instruction  covering  PR  package 
contents  and  preparation; 

3)  failure  to  involve  contracting 
personnel  early  in  the  acquisition 
cycle;  and, 

4)  lack  of  understanding  of  the  source 
selection  process. 

These  problems  were  corrected  by  returning  the  package 
to  the  originating  office  with  extensive  comments  on 
content  and  format,  modifying  the  contract  after  award 
to  correct  errors  and  provide  missing  information,  and 
holding  team  meetings  and  reviews  to  resolve  the 
issues.  Suggestions  to  avoid  the  problem  in  the  future 
included  earlier  involvement  by  contracting  personnel 
in  acquisition  planning  and  the  requirements 
development  process,  development  of  a  detailed 
operating  instruction  on  PR  package  requirements,  and 
education  on  the  source  selection  process. 

The  fourth  problem  characterized  as  major  involved 
a  PR  package  which  did  not  accurately  reflect  the 
desired  program,  resulting  in  the  receipt  of  proposals 
which  exceeded  the  program  budget.  The  PR  package  had 
included  excessive  requirements,  such  as  full  cost 
reporting  on  a  firm  fixed  price  contract,  and  included 
requirements  which  were  not  properly  tailored  for  the 
program.  The  problem  was  solved  by  de-scoping  the 
effort  to  delete  excessive  or  inappropriate 
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requirements  and  re-soliciting.  The  suggested  long¬ 
term  solution  was  to  ensure  that  requirements  are 
appropriately  tailored  to  the  program;  no  means  of 
ensuring  such  tailoring  was  suggested. 

Both  subjects  who  cited  moderate  problems  with  PR 
package  development  had  experienced  problems  with  the 
requirements  definition  process.  Requirements  for  the 
program  were  not  defined  well  early  in  the  process  and 
continued  to  change  throughout  the  source  selection. 
One  of  these  respondents  stated  that  translating  the 
general  requirements  of  the  Statement  of  Need  into  the 
detailed  requirements  of  a  Statement  of  Work  and 
specifications  was  very  difficult  and  frequently  led 
to  conflicts  between  the  engineers  and  users.  The 
cited  causes  were: 

1)  a  lack  of  understanding  of  mission 
requirements  on  the  part  of  those 
responsible  for  writing  the 
detailed  requirements  documents; 

2)  that  users  frequently  have  a  particular 
equipment  item  or  system  in  mind, 
while  the  engineers  in  the  program 
office  have  no  "vested  interest'  in 
any  system;  and, 

3)  conflict  between  the  users  and  the 
acquisition  community  due  to  users’ 
lack  of  knowledge  of  acquisition 
procedures  and  regulations. 


55 


The  problems  were  handled  by  making  changes  to  the 
Statement  of  Work  and  specifications  throughout  the 
source  selection  and  after  contract  award,  and  through 
persistence  by  engineering  and  contracting  personnel  to 
overcome  the  users'  pre-existing  product  interests. 
Long-term  solutions  were  to  define  the  requirements 
early  in  the  process,  to  ensure  greater  familiarity 
with  mission  requirements,  and  to  mitigate  the  effects 
of  politics  in  the  system,  which  can  cause  personnel  to 
push  the  'party  line'  rather  than  being  realistic  about 
what  can  be  achieved  technically. 

Both  minor  problems  concerned  incomplete  PR 
packages  and  the  failure  of  the  program  office  to 
provide  the  packages  to  the  contracting  office  in  a 
timely  manner.  Lack  of  knowledge  and  a  definitive 
standard  for  PR  packages  were  cited  as  the  cause. 
Discussions  with  the  drafters  of  the  packages  were 
held  to  resolve  the  problems.  The  suggested  long-term 
solution,  in  both  cases,  was  the  development  of 
internal  operating  instructions  defining  the  contents 
of  a  PR  package. 

Analysis .  Development  of  the  PR  package 
represents  an  area  where  significant  improvement  could 
be  made  in  the  source  selection  process;  eight  of  the 
twelve  respondents  identified  problems  with  this 
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activity.  Because  PR  packages  and  their  problems  are 
not  unique  to  source  selections,  improvements  in  this 
area  could  benefit  all  types  of  acquisitions. 

In  addition  to  the  previously  cited  need  for 
formal  education  on  the  source  selection  process,  the 
respondents  also  cited  the  need  for  a  standard  defining 
required  PR  package  contents.  The  researcher  is  aware 
that  some  contracting  directorates  at  ASD  have 
developed  operating  instructions  which  address  PR 
package  contents;  other  directorates  could  easily 
develop  such  standards  to  alleviate  this  problem. 

An  additional  factor  is  the  lack  of  quality 
control  for  PR  packages.  If  adequate  tecnnical 
reviews  of  the  completed  packages  were  accomplished 
before  the  PR  packages  left  the  program  office,  the 
number  of  inadequate  packages  should  be  reduced. 

The  problem  with  requirements  definition  cited  by 
two  respondents  is  not  so  easily  dealt  with.  This 
problem  has  been  cited  by  critics  of  the  defense 
acquisition  system  (8:145-147;  9:107-117).  If 
politics  could  be  removed  from  the  requirements  * 
definition  process,  this  problem  would  be  alleviated; 
however,  reforms  to  the  methods  of  requirements 
definition  are  beyond  the  scope  of  this  study. 
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Request  for  Proposal  (RFP)  preparation  and 


review.  (Major  -  2;  Moderate  -  3;  Minor  -  1) 

Responses .  Both  major  problems  were  related 
to  the  RFP  review  process.  One  respondent  cited  the 
late  receipt  of  review  comments  from  AFSC ,  which 
necessitated  an  extension  to  the  proposal  due  date  to 
accomodate  the  comments.  This  problem  was  caused  by  a 
’business  as  usual*  review  of  the  RFP  by  AFSC.  The 
problem  was  handled  by  extending  the  due  data  for 
proposals  to  answer  or  implement  tho  AFSC  comments. 

The  suggested  solution  was  the  establishment  of  a  time 
limit  for  AFSC  review  of  RFPs . 

The  second  major  problem  involved  the  number  of 
reviews  required.  For  a  high  dollar  program,  the 
following  reviews  are  required: 

1)  Acquisition  Review  Team  (ART)  - 
require  approximately  one  month 
including  preparation,  briefing, 
reviews,  response  to  comments, 
and  resolution  of  problems; 

2)  Legal  review  by  JAG  -  requires 
three  to  five  days,  minimum; 

3)  Contracting  directorate  review  - 
requires  three  to  five  days 
minimum; 

4)  Contract  review  committee--  requires 
five  days  minimum;  and, 

5)  AFSC  contract  review  committee  - 
review  is  accomplished  after  release 
of  RFP  and  may  be  waived  if  AFSC 
workload  is  heavy.  The  lack  of  a 
review  on  the  RFP  can  cause  problems 
at  contract  award  after  a  signed 
contract  has  been  received  from  the 
contractor . 
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Those  multiple  reviews  can  result  in  conflicting 
opinions  which  must  be  resolved  by  the  contracting 
office,  often  resulting  in  constant  changes  to  all 
parts  of  the  RFP.  The  cited  cause  of  this  problem  was 
related  to  the  recent  streamlining  initiatives,  which 
have  shortened  the  time  available  for  activities 
between  release  of  the  RFP  and  source  selection 
decision,  but  added  to  the  time  required  for  up-front, 
unmonitored  activities  by  adding  requirements  like  the 
ART.  In  addition,  mandates  by  the  SSA  to  award 
without  discussions  caused  reviewers  to  be  over¬ 
cautious,  lengthening  the  time  required  for  the 
reviews.  Under  extremely  unusual  circumstances, 
certain  reviews  may  be  waived;  generally,  however,  the 
only  choice  for  dealing  with  this  problem  was  to 
accomplish  all  the  required  reviews  and  resolve  all  the 
review  comments.  The  suggested  solution  for  this 
problem  called  for  allowing  the  ART  to  serve  as  the  sole 
RFP  review  at  ASD.  All  participants  in  the  current  ASD 
review  process  participate  in  the  ART;  therefore,  when 
the  ART  is  complete,  all  parties  could  sign  off  on  the 
RFP  at  that  time  to  complete  the  ASD  review  cycle  in 
one  step. 

All  the  moderate  problems  cited  concerned  the 
time  required  to  accomplish  the  preparation  and  review 
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process.  The  causes  included  buyer  workload, 
constantly  changing  regulatory  requirements,  excessive 
number  of  reviews,  and  excessive  time  required  to  have 
RFPs  reproduced.  These  causes  are  hard  to  influence; 
therefore,  the  respondents  had  dealt  rather 
unsuccessfully  with  these  problems  in  the  past.  As 
long-term  solutions,  the  respondents  suggested  sharing 
workload  among  groups,  requesting  priority  reviews  and 
priority  printing,  overtime,  and  cutting  the  number  of 
reviews  required. 

The  minor  problem  concerned  the  lengthening  of 
the  RFP  preparation  cycle  by  the  streamlining 
initiatives;  these  initiatives  move  time  from  the 
portion  of  the  schedule  which  is  tracked  to  the 
preparation  period,  require  additional  document 
reviews  and  approvals,  and  add  new  higher  level 
coordination/approval/brief  mgs .  The  respondent  was 
unable  to  identify  an  associated  cause.  The  problem 
has  been  mitigated  through  overtime,  dedication  of 
personnel  to  source  selections,  and  letting  other 
workload  slip.  Suggested  long-term  solutions  included 

1)  cutting  the  number  and  levels  of  review, 

2)  delegation  of  approvals  to  the  lowest 
appropriate  level; 

3)  better  cooperation  between  team 
members  and  users;  and, 

4)  establishment  of  generic  samples 

of  various  source  selection  documents 
to  serve  as  models. 
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Analysis .  Half  of  the  respondents  identified 


problems  with  RFP  preparation  and  review;  all  problems 
related  to  the  time  required  to  complete  the  activity. 
Five  of  the  six  respondents  characterized  the  number  of 
required  reviews  as  excessive.  The  suggestion  that  the 
ART  serve  as  the  sole  RFP  review  at  ASD  appears  to  be  a 
practical  means  of  avoiding  redundant  reviews  and 
saving  time.  Implementation  of  this  change  would 
require  revision  of  the  ASD  FAR  Supplement. 

One  very  interesting  point  arose  during  the 
discussions  of  RFP  preparation  and  review  problems. 
Several  respondents  cited  the  recent  streamlining 
initiatives  as  the  cause  of  increasing  RFP  preparation 
leadtimes.  The  respondents  cited  additional 
requirements  imposed  by  the  initiatives  which  are 
designed  to  minimize  the  time  consumed  by  source 
selections.  This  topic  will  be  addressed  in  greater 
detail  in  the  section  of  this  chapter  which  deals  with 
the  general  comments. 

Receipt  of  proposals.  (Minor  -  1).  One  minor 
problem  was  identified  related  to  receipt  of  proposals. 
This  was  an  administrative  problem  related  to  security 
issues  specific  to  the  office  in  which  the  respondent 
was  assigned;  the  problem  is  unrelated  to  the  source 
selection  process. 
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Quick  Look  briefing  or  memo.  (Minor  -  1) 


Responses .  The  respondent  stated  that  having 
to  stop  the  entire  process  to  prepare  this  memo  was  a 
minor  problem.  The  cause  lay  in  the  perceived  lack  of 
value  of  the  memo,  which  relays  very  little  information 
to  the  SSA.  Although  the  respondent  has  routinely 
processed  the  required  memo  in  the  past,  he  suggested 
that  the  requirement  for  the  memo  be  deleted. 

Analysis .  This  activity  represents  only  a 
minor  problem  in  the  process.  However,  because  the 
Quick  Look  memo  does  provide  very  little  benefit  to  the 
process,  consideration  should  be  given  to  changing  the 
requirement  to  allow  a  notification  to  the  SSA  of  the 
proposals  received,  rather  than  requiring  a  briefing  or 
memo . 

Technical  evaluation.  (Major  -  9;  Moderate  -  4; 
Moderate  to  Minor  -  2;  Minor  -  4) 

Responses .  One  major  problem  experienced 
with  the  technical  evaluation  was  the  lack  of  specific 
information  and  support  in  the  technical  evaluation 
report,  caused  by  lack  of  training.  The  problem  was 
resolved  by  holding  discussions  with  the  area  chiefs 
until  adequate  rationale  was  obtained  for  the  positions 
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stated  in  the  evaluation  report.  The  respondent 
suggested  formal  training  and  an  update  of  the  ASD/PMPS 
source  selection  handbook  as  long-term  solutions. 

The  remaining  major  problems  dealt  with  two 
different  aspects  of  technical  evaluation: 

»)  Clarification  Requests  (CRs) ,  Deficiency  Reports 
(DRs) ,  and  Modification  Requests  (MRs) ,  and  2)  effects 
of  streamlining  initiatives.  Two  respondents  cited  the 
large  number  of  CRs  and  DRs  that  were  required,  which 
delayed  completion  of  the  evaluation.  In  one  case,  the 
problem  was  caused  by  deficiencies  in  the  RFP;  the 
second  was  caused  by  the  failure  of  the  offerors  to 
comply  with  RFP  requirements.  Because  the  information 
was  needed  to  complete  the  evaluations,  the  CRs  and  DRs 
were  processed.  Solutions  included  more  thorough 
reviews  of  the  technical  requirements  documents  in  the 
RFP  and  better  communication  with  potential  offerors  to 
ensure  that  RFP  requirements  are  understood  and 
addressed . 
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Another  problem  concerned  the  discovery  during  the 
technical  evaluation  that  an  important  program 
requirement  had  been  omitted  from  the  RFP ,  requiring 
the  issuance  of  an  MR  to  incorporate  it,  and  delaying 
the  contract  award  by  two  weeks.  The  suggested 
solution  included  a  thorough  technical  review  of  the 
RFP  and  better  planning. 

In  another  case,  numerous  ‘re-hashes'  of  the  CRs 
and  DRs  were  required  to  reduce  the  number,  which  made 
the  process  very  time  consuming.  The  cited  cause  was  a 
goal,  established  by  the  ASD  Commander,  of  zero  CRs  and 
DRs.  The  respondent  felt  that  this  goal  caused  a  loss 
of  support  from  the  evaluators  who  felt  that  their 
comments  were  being  ignored  and  that  management  was 
attempting  to  control  the  issues  to  be  addressed 
through  controlling  the  CR/DR  process.  The  problem  was 
handled  by  deferring  issues  to  the  discussion  period, 
rather  than  resolving  them  during  the  evaluation.  The 
proposed  long-term  solution  included  education  and  a 
change  in  the  climate  to  allow  proper  handling  of 
issues  which  must  be  resolved  before  contract  award. 

The  major  problems  related  to  the  streamlining 
initiatives  fell  into  two  categories.  The  first  had  to 
do  with  constraints  placed  on  the  evaluation  because  of 
streamlining.  One  respondent  stated  that  the 
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accelerated  schedules  do  not  allow  adequate  time  to 
conduct  a  thorough  review  of  the  proposals.  The  page 
limits  on  proposals  were  also  cited  as  a  factor 
limiting  the  adequacy  of  the  evaluations.  Although  the 
subjects  complied  with  the  constraints  mandated  by  the 
streamlining  initiatives,  they  believed  that  these 
constraints  severely  limited  the  usefulness  of  the 
resulting  evaluations.  They  suggested  that  sufficient 
time  and  information  be  allowed  to  ensure  a  meaningful 
evaluation.  One  respondent  suggested  that,  if  adequate 
time  and  data  cannot  be  provided,  the  requirement  for 
technical  evaluation  should  be  deleted  because  it 
provides  no  meaningful  measure. 

The  final  major  problem  was  caused  by  a  mandate  by 
the  SSA,  prior  to  the  start  of  the  source  selection, 
that  award  would  be  made  without  discussions.  This 
mandate  meant  that  the  SSA  would  not  approve  issuance 
of  any  CRs  or  DRs ,  which  took  the  evaluation  out  of  the 
hands  of  the  evaluators.  The  evaluators  were  forced  to 
defend  (unsuccessfully)  their  need  for  information,  had 
valid  concerns  which  could  not  be  addressed,  and  were 
unable  to  obtain  clarifications  which  were  needed  for 
an  unrestricted  evaluation.  The  respondent  felt  that 
the  mandate  for  award  without  discussions  severely 
limited  the  validity  and  completeness  of  the 
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evaluation . 


The  SSA's  mandate  was  complied  with,  and 
the  contract  was  awarded  without  discussions.  The 
respondent  proposed  as  a  long-term  solution  that  the 
CR/DR  process  be  taken  from  the  hands  of  the  SSA  and 
returned  to  the  evaluation  team  chairperson  to  ensure 
that  complete  evaluations  are  performed  and  that  the 
government  can  obtain  the  benefits  that  can  be  derived 
from  meaningful  discussions. 

The  moderate  and  moderate  to  minor  problems 
involved : 

1)  the  desire  of  evaluators  to  send 
CRs  for  ‘nice  to  have*  information 
which  was  not  required  and  could 
not  be  evaluated; 

2)  thf  need  for  significant  revisions 
to  CRs,  DRs ,  close-out  sheets, 
and  evaluation  reports  submitted 
by  the  evaluators; 

3)  the  requirement  for  SSA  approval 
of  CRs  and  DRs; 

4)  untimely  receipt  of  technical 
evaluation  reports;  and, 

5)  excess  time  spent  in  training 
during  the  evaluation. 

The  causes  cited  for  these  problems  were: 

1)  lack  of  experience  and  education; 

2)  lack  of  technical  skill; 

3)  lack  of  awar'eness  on  the  part 
of  the  evaluators  concerning 
contracting  requirements  and 

how  contract  awards  are  justified; 
and , 

4)  lack  of  training. 

The  situations  were  handled  through  close  teamwork  and 
meetings.  The  respondents  cited  the  need  for  formal 
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training.  It  was  a.lso  suggested  that  approval  of  CRs 
and  DRs  be  delegated  to  a  lower  level  than  the  SSA. 

One  respondent  proposed  that  experienced  source 
selection  teams  be  formed  in  each  Deputy;  these  teams 
would  ue  supplemented  by  technical  experts  to  conduct 
all  the  source  selections  for  that  organization. 

One  minor  problem  involved  the  need  to  issue  an 
MR  to  correct  errors  in  the  technical  requirements 
which  were  discovered  during  the  evaluation,  thus 
adding  to  the  length  of  the  source  selection.  The  MR 
was  needed  because  the  technical  requirements  documents 
were  prepared  too  quickly  and  were  not  adequately 
reviewed  by  technical  personnel.  The  MR  corrected  the 
errors,  but  a  formal  review  of  the  Statement  of  Work, 
specifications,  and  data  requirements  would  have 
ensured  that  the  documents  were  correct  as  submitted  to 
the  contracting  office. 

Another  minor  problem  concerned  the  Performance 
Risk  Analysis  (PRA) .  The  respondent  stated  that  the 
PRA  was  too  general  and  lacking  in  detail;  the  grades 
given  for  past  performance  were  too  high;  and  that  too 
much  emphasis  was  placed  on  past  performance  in  the 
evaluation,  to  the  point  that  past  performance  became 
the  most  important  issue  in  contractor  selection.  The 
problems  with  the  PRA  were  resolved  through  discussions 
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between  the  contracting  officer  and  evaluators.  The 
proposed  long-term  solution  included  clarification  of 
the  PRA  content  requirements  and  reduced  emphasis  on 
past  performance. 

An  additional  minor  problem  was  that  the 
evaluators  did  not  know  the  difference  between  a  CR  and 
a  DR,  resulting  in  increased  time  required  for  review 
and  revision  of  the  CR/DR  submittals.  The  stated  cause 
was  lack  of  understanding,  and  the  suggested  solution 
was  training. 

Analysis .  Technical  evaluation  and  its  sub- 
activities  received  19  problem  ratings.  The  extent  of 
the  problems  experienced  with  technical  evaluations 
indicates  that  this  activity  provides  the  greatest 
opportunity  for  improvement  in  the  process. 

The  majority  of  the  problems  described  resulted 
from  the  lack  of  education  or  experience  of  evaluation 
team  members.  The  technical  teams  were  unable  to 
produce  acceptable  documentation  in  a  timely  manner. 
Providing  adequate  training  prior  to  the  evaluation 
period  would  improve  the  quality  of  the  evaluation 
documents  and  reduce  the  time  needed  to  review  and 
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revise  tho  documentation.  A  good  understanding  of  the 
process  would  also  reduce  the  amount  of  non-relevant 
data  required  from  offerors,  providing  a  reduction  in 
time  and  possible  minor  cost  savings. 

The  suggestion  that  lack  of  experience  be  overcome 
by  establishing  expert  source  selection  teams  in  each 
Deputy  sounds  attractive,  but  does  not  seem  very 
practical  due  to  the  limited  number  of  source 
selections  conducted  in  most  organizations.  A 
variation  of  this  idea  will  be  addressod  in  the  section 
of  this  chapter  which  presents  the  general  comments 
made  during  the  interviews. 

The  second  most  frequently  identified  cause  of 
problems  was  the  streamlining  initiatives,  which 
limit  the  scope  of  the  evaluations.  Because  technical 
excellence  is  often  the  most  important  element  in 
determining  the  successful  offeror,  any  limitation  on 
the  validity  and  completeness  of  the  technical 
evaluations  seems  undesireable ;  thorough  evaluations 
are  necessary  to  provide  the  highest  probability  that 
the  resultart  contract  will  provide  a  system  tha-t  meets 
the  users'  needs.  A  reduction  in  the  time  required  to 
conduct  source  selection?  may  not  be  a  satisfactory 
pay-off  if  the  price  for  that  reduction  is  a  compromise 
in  the  quality  of  the  end  item.  The  researcher  would 
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take  greatest  exception  to  the  apparent  elimination  of 
DRs ,  which  appears  to  be  a  violation  of  the  FAR 
requirement  to  disclose  and  allow  correction  of 
deficiencies  prior  to  contract  award.  Even  if 
deficiencies  were  corrected  through  modifications  after 
award  to  protect  the  quality  of  the  equipment,  it  would 
appear  that  the  contractor  would  stand  to  gain 
financially  through  equitable  adjustments  for  changes 
which  should  have  included  in  the  scope  and  cost  of  the 
basic  contract.  It  would  seem  to  be  in  the 
government's  best  interests  to  reverse  the  trend  toward 
eliminating  correction  of  proposal  deficiencies  for  the 
sole  purpose  of  avoiding  discussions  and  BAFOs . 

Several  of  the  technical  evaluation  problems 
resulted  from  inadequacies  in  the  technical 
requirements  documents  in  the  RFPs.  This  problem  is 
a  result  of  the  inadequate  quality  control  for  PR 
packages,  which  was  previously  discussed. 

Mid-Term  Briefing.  (Moderate  -  2;  Minor  -  1) 

Responses .  Both  moderate  problems  concerned 
the  inordinate  amount  of  time  required  -to  prepare  for 
the  briefing,  including  chart  preparation  and  dry  runs. 
In  one  case,  the  effort  expended  was  wasted,  in  that 
the  briefing  was  conducted  unsuccessfully  and  had  to  be 
completely  revised  and  re-accomplished.  This 
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respondent  believed  that  the  failure  of  the  briefing 
was  caused  by  the  lack  of  knowledge  of  content  and 
format  requirements  and  lack  of  understanding  of  the 
purpose  of  the  briefing;  the  suggested  long-term 
solution  was  education.  The  second  respondent  simply 
felt  that  the  effort  required  to  produce  the  briefing 
was  far  in  excess  of  the  benefit  derived  from  the 
briefing,  and  suggested  that  the  requirement  for  a  Mid- 
Term  briefing  be  deleted. 

The  minor  problem  concerned  the  information  to  be 
presented  at  the  Mid-Term.  The  Mid-Term  presantM  the 
evaluation  of  the  "as  received"  proposals;  no 
information  on  the  resolution  of  DRs  is  presented.  The 
respondent  had  problems  *-  *h  evaluators  wishing  to 
present  information  on  issues  which  had  been  resolved. 
This  was  caused  by  the  natural  reluctance  of  people  to 
present  out-of-date  information.  The  problem  was  dealt 
with  through  numerous  reminders  and  references  to 
regulatory  guidance  which  defines  the  information  to  be 
included  in  the  Mid-Term.  The  proposed  long-term 
solution  was  to  allow  the  information  co  be  presented 
as  part  of  the  Mid-Term  briefing. 


71 


Analysis .  Preparation  and  conduct  of  the 


Mid-Term  briefing  requires  a  considerable  effort  from 
the  entire  source  selection  team.  It  would  seem  that 
this  effort  could  be  expended  in  more  beneficial 
activities  if  the  scope  of  the  briefing  were  changed. 
The  Mid-Term  presents  great  detail  about  the  initial 
technical  evaluation  of  the  'as  received*  proposals; 
this  information  is  obsolete  after  resolution  of  the 
CRs/DRs.  It  would  seem  reasonable,  therefore,  to 
present  only  an  overview  of  the  evaluation  results  and 
the  status  of  any  significant  problems  encountered.  In 
many  cases,  it  would  seem  that  a  memorandum  could  be 
sent  to  the  SSA  in  lieu  of  a  formal  presentation. 

A  brief  overview  or  memorandum  would  keep  the  SSA 
informed  of  the  status  of  the  source  selection,  but 
would  relieve  the  administrative  burden  of  a  detailed 
presentation  of  information  which  is  of  temporary 
value.  Changes  to  the  ASD  supplements  to  AFR  70-15 
and  AFR  70-30  would  be  required  to  modify  the  Mid- 
Term  briefing  requirements. 
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estimate .  (Major  -  3;  Moderate  -  1) 

Responses .  The  difficulty  and  time  involved 
in  obtaining  the  needed  information  from  the 
Administrative  Contracting  Officer  (AGO)  and  the 
Defense  Contract  Audit  Agency  (DCAA)  were  cited  by  two 
respondents  as  a  major  problem  involved  in  cost/price 
analysis.  Often  the  audits  caused  significant  delays, 
and  frequently  DCAA  did  full  a-  ts  on  the  prime 
contractor  and  all  proposed  subcontractors  when  only 
limited  audits  on  the  prime  had  been  requested.  St&ted 
causes  for  the  problem  included  the  absence  of 
priorities  at  DCAA;  the  lack  of  personal  involvement  by 
field  personnel,  leading  to  a  lack  of  the  closest 
attention  to  requests  for  information;  and  failure  of 
jutside  agencies  to  become  attuned  to  tile  Air  Force's 
accelerated  schedules.  The  immediate  solutions  to  the 
problem  included  frequent  telephone  follow-ups  and 
close  monitoring  of  audit  schedules,  and  elevation  of 
delinquencies  to  upper  management  for  attempted 
resolution;  in  many  cases,  however,  attempts  to 
accelerate  audits  were  unsuccessful.  One  respondent 
suggested  that  DoD  implement  initiatives  to  make  field 
personnel  more  responsive  to  buying  agency  requests  for 


information . 


Another  major  problem  with  cost/price  analysis 
involved  the  role  of  program  control  (cost  estimating) 
personnel  in  the  analysis.  The  respondent  found  that 
program  control  personnel  believed  that  tl<ey  were 
responsible  for  performing  the  analysis  which  results 
in  the  determination  of  ‘fair  and  reasonable"  prices. 
(Per  the  Federal  Acquisition  Regulation,  this 
responsibility  belongs  to  the  contracting  officer,  who 
may  be  assisted  by  a  price  analyst.)  This  confusion 
led  to  a  duplication  of  effort,  in  that  program  control 
personnel  obtained  information  and  performed  an 
analysis  which  was  never  used,  because  the  official 
price  analysis  was  performed  by  the  contracting 
officer  and  price  analyst.  The  cause  of  this  problem 
was  a  lack  of  understanding  by  program  control 
personnel  of  the  functions  of  the  contracting  officer 
vs .  program  control.  The  resolution  of  the  problem  was 
very  difficult  because  program  control  was  not  willing 
to  limit  their  activity  to  what  was  desired  by  the 
contracting  officer,  which  led  to  a  significant  waste 
of  resources.  The  long-term  solution  was  to  clarify  to 
program  control  personnel  the  responsibilities  of  the 
contracting  officer. 
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Two  respondents  also  found  major  problem  with  the 
Most  Probable  Cost  (MPC)  estimate.  One  respondent 
stated  that  excessive  data  was  required  to  be  submitted 
for  performance  of  the  MPC.  It  was  also  felt  that  the 
computer  models  used  by  program  control  to  perform  the 
MPC  were  frequently  inappropriate,  resulting  in  an 
invalid  MPC.  The  stated  cause  was  the  unfamiliarity  of 
cost  estimating  personnel  with  the  MPC  process,  and  the 
reliance  on  computer  models  which  are  not  appropriately 
tailored  to  individual  programs.  The  problem  related 
to  excess  data  requirements  was  dealt  with  by  holding 
meetings  between  the  directorates  to  negotiate 
reductions  in  the  data  submittal  requirements.  No 
satisfactory  alternative  to  using  the  inappropriate 
computer  model  was  achieved  or  suggested. 

The  second  respondent  stated  that  the  MPC  was  a 
very  time-consuming  process  which  resulted  in  no 
benefit  to  the  Air  Force.  This  problem  derived  from 
the  nature  of  price  determination  on  competitive 
acquisitions;  the  best  cost/price  is  determined  by 
conpetition.  The  requirement  for  an  MPC  at  certain 
dollar  values  'assumes  that  the  MPC  developer  knows 
more  than  the  competitive  system."  Even  if  the  MPC  is 
at  variance  from  the  offeror's  proposed  price,  the  Air 
Force  can  only  point  out  the  general  areas  of 
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difference,  and  has  no  power  to  make  the  offeror  change 
the  proposed  cost  to  comply  with  the  MPC  estimate.  The 
respondent  has  complied  with  the  requirement  for 
conduct  of  an  MPC,  but  suggested  that  the  requirement  be 
deleted  as  unnecessary  in  a  competitive  environment. 

The  moderate  problem  identified  with  cost/price 
analysis  also  related  to  the  nature  of  price 
determination  in  a  competitive  environment.  The 
respondent  questioned  the  extent  of  analysis  required 
when  prices  are  determined  through  competition  vs . 
negotiation.  The  respondent  felt  that  too  much 
analysis  is  done  to  no  avail  in  the  competitive 
environment  of  source  selections.  The  respondent 
stated  that  such  extensive  analysis  is  performed  as  an 
aid  in  explaining  the  ‘fair  and  reasonable"  nature  of 
the  final  price  because  the  organizational  climate  has 
not  yet  adapted  to  accepting  competition  as  the 
determinor  of  price.  The  respondent  has  performed 
limited  analysis  in  the  past,  verifying  rates  and 
factors  through  the  field  and  using  the  technical 
evaluation  to  justify  hours.  The  suggested  long-term 
solution  was  to  severely  limit  analysis  and  rely  more 
heavily  on  competition  to  justify  prices  as  "fair  and 
reasonable . " 
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Analysis .  One  third  of  the  subjects  have 


experienced  significant  problems  with  cost/price 
analysis  or  the  MPC.  The  most  serious  of  these 
problems  call  into  question  the  need  for  these 
activities.  In  a  competitive  environment,  it  should  be 
necessary  to  perform  only  enough  analysis  to  assure  the 
government  that  the  offeror  has  submitted  a  complete 
cost  proposal  which  realistically  reflects  the  total 
requirement  of  the  solicitation.  Extensive  cost 
analyses  or  MPC  estimates  are  of  very  limited  value 
when  prices  are  determined  through  BAKGs  rather  than 
negotiation.  Limiting  the  cost  analysis  performed  for 
source  selections  is  possible  under  recent 
interpretations  of  ‘adequate  price  competition"  but  is 
not  widely  accepted,  perhaps  because  the  policy  is  new 
and  not  well  defined.  Elimination  of  the  MPC 
requirement  could  be  accomplished  through  revision  of 
AFR  70-15.  Increased  use  of  limited  cost  analysis  and 
elimination  of  the  MPC  would  alleviate  the  other 
pr  blems  cited  by  the  respondents  for  this  activity. 

Model  contract  preparation.  (Major  -  3; 

Moderate  to  major  -  1;  Moderate  -  1;  Minor  -  1) 

Responses .  One  respondent  identified  a  major 
problem  in  contract  preparation  which  was  related  to 
the  use  of  MIL-PRIME  specifications.  With  MIL-PRIME, 


77 


each  offeror  has  a  different  Statement  of  Work  (SOW) 
based  on  each  individual  approach  to  the  program.  The 
problem  arose  from  SOW  changes  resulting  from  a 
Modification  Request.  Rather  than  writing  the  changes 
themselves,  the  government  engineers  had  the  offerors 
submit  revised  SOWs .  The  engineers  did  not  review  the 
SOW  changes,  and  the  contracting  office  later 
discovered  that  the  changes  had  not  been  made  as 
desired.  The  engineers  had  tried  to  save  time,  without 
considering  the  possible  consequences;  this  situation 
resulted  from  lack  of  experience  and  guidance.  The 
contract  negotiator  explained  the  problems  to  the  team 
members,  and  worked  with  the  team  to  resolve  them, 
resulting  in  time  which  should  have  been  spent  on 
contracting  activities  being  expended  in  accomplishing 
the  tasks  of  other  functional  team  members.  The 
suggested  long-term  solution  included  placing  more 
emphasis  on  the  assignment  of  team  members  to  ensure 
that  any  lack  of  experience  is  offset  by  experienced 
participants,  providing  close  guidance  for 
inexperienced  members,  and  re-examining  the  use  of  MIL- 
PRIME  in  source  selections. 

The  second  major  problem  was  the  fact  that  the 
physical  preparation  of  the  model  contracts  was  very 
time-consuming  and  labor  intensive.  In  this  instance, 
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the  lack  of  experienced  clerical  personnel  required  the 
contract  negotiators  to  perform  a  major  portion  of  the 
clerical  effort.  The  documents  were  typed  by  the 
contract  negotiator,  who  was  required  to  work  a  great 
deal  of  overtime  to  complete  this  clerical  task.  The 
suggested  long-term  solution  was  to  ensure  that 
adequate,  experienced,  dedicated  clerical  support  is 
available  for  source  selections. 

The  final  major  problem  involved  the  accidental 
deletion  of  an  evolving  model  contract  from  the 
contract  writing  data  base.  While  this  was  a 
significant  problem  for  the  contracting  team  in  this 
case,  it  was  caused  by  human  error  unrelated  to  the 
source  selection  process. 

The  remaining  problems  in  model  contract 
preparation  all  involved  the  continuous  flow  of 
contract  changes  required,  either  due  no  CRs  and  DRs  or 
to  changes  in  the  FAR  and  its  supplements,  and  the 
difficulty  of  accomodating  them,  with  quality,  under 
accelerated  contracting  schedules.  In  one  case,  the 
contract  was  being  awarded  under  a  mandate  of  'no 
discussions , *  which  precluded  the  incorporation  of  FAR 
changes.  The  other  cases  involved  late  receipt  of 
CR/DR  changes.  In  all  three  cases,  the  solutions 
involved  modification  of  the  contracts  after  award. 
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Tho  suggested  long-term  solutions  included  providing 
more  contracting  manpower;  establishing  deadlines  for 
receipt  o f  technical  changes  in  the  contracting  office; 
allowing  for  the  provision  of  a  notice  to  the 
contractor  that  FAR  changes  will  be  included  in  the 
resultant  contract,  without  such  notice  being 
considered  to  be  discussions;  and  clarification  by 
upper  management  as  to  priorities  -  *what  is  more 
important,  quality  or  schedule?* 

Analysis .  Half  the  samplo  identified 
probloms  with  model  contract  preparation.  The  most 
significant  factor  affecting  this  activity  is  the 
labor-intensive  nature  of  the  contract  preparation 
process  and  the  shortage  of  clerical  personnel.  The 
frequent  FAR  and  FAR  supplement  changes  exacerbate  the 
situation.  When  the  CR/DR/MR  change  process  is  not 
adequately  controlled,  contract  preparation  becomes 
very  burdensome.  This  is  particularly  true  when  time 
pressures  are  present.  Contract  preparation  requires 
the  cooperation  of  all  team  members  to  minimise  changes 
and  ensure  the  accuracy  of  contract  documents. 
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The  request  for  management  clarification  of  the 
priorities  to  be  placed  on  quality  and  schedule 
indicates  that  the  streamlining  initiatives  have  also 
influenced  the  contract  preparation  process.  It 
appears  that  quality  is  no  longer  the  major  concern. 

Discussions .  (Moderate  to  major  -  1) 

Responses .  In  this  instance,  the  contracting 
officer  was  constrained  by  management  in  the  subjects 
to  be  addressed  during  discussions.  In  addition  to 
attempting  to  limit  the  scope  of  the  discussions, 
manage  also  required  the  contracting  officer  to 

limit  .  .  cten  information  provided  to  the  offerors  in 
preparation  for  the  discussions  to  the  general  areas 
for  discussion,  rather  than  providing  the  specific 
questions  to  be  addressed.  The  respondent  was  unable 
to  cite  a  specific  cause  for  this  situation,  but  felt 
that  it  was  in  some  way  related  to  the  Commander’s  goal 
of  zero  C'Rs  and  DF.s .  The  immediate  solution  required 
the  contracting  officer  to  hold  extensive  telephone 
discussions  with  the  offerors  in  advance  of  the 
discussions,  to  ensure  that  all  parties  were  pre-pared. 
The  management  direction  to  limit  the  scope  of 
discussions  was  ignored,  because  the  team  felt  that 
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certain  issues  had  to  be  addressed.  The  suggested 
solution  was  to  remove  the  outside  influences  on  the 
contracting  officer,  so  that  the  contracting  officer 
can  determine  the  nature  of  the  discussions. 

Analysis .  This  problem  appears  to  be  an 
outgrowth  of  the  streamlining  initiatives;  limiting  the 
discussions  would  accelerate  the  schedule.  However, 
this  acceleration  could  come  at  the  expense  of  quality 
and  cost,  similar  to  the  situation  discussed  under 
'technical  evaluation*  above. 

BAFO  process.  (Major  -  1;  Moderate  -  1; 

Minor  -  2) 

Responses .  The  major  problem  involved  the 
respondent's  perception  that  use  of  BAFOs  is  out  of 
favor,  and  that  award  without  discussions  may  be 
mandated  by  the  SSA  prior  to  the  start  of  the  source 
selection.  The  respondent  stated  that  being  required 
to  award  without  discussions  arbitrarily  takes  the 
decision  concerning  the  need  for  a  BAFO  out  of  the 
contracting  officer's  hands  and  "perverts'  the  source 
selection  process.  The  cited  cause  for- this  problem 
was  the  current  emphasis  on  schedule  over  fairness  and 
logic,  which  the  respondent  assumed  to  be  a  local 
interpretation  of  the  AFSC  Commander’s  concerns  on 
source  selection  streamlining.  The  respondent  stated 
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that  past  o£ forts  to  deal  with  this  problem  had  boon 
unsuccessful.  Attempts  had  been  made  to  communicate 
through  the  SSAC  that  a  pro-do  termination  to  award 
without  discussions  and  BAFO  was  not  an  optimal 
situation;  these  attempts  were  not  accepted.  The 
proposed  long-term  solution  was  abolition  of  the 
existing  policy  of  mandated  award  without  BAFO.  The 
respondent  stated  that  the  circumstances  of  each 
individual  acquisition  must  be  examined  before  an 
appropriate  determination  concerning  discussions  and 
BAFO  can  be  made.  It  was  suggested  that  ASD/PM  should 
make  every  attempt  to  get  this  policy  changed. 

The  modorato  problem  related  to  cases  in  which  the 
SSA  has  not  mandated  that  award  will  be  made  without 
discussions.  In  those  cases,  it  was  virtually 
impossible  to  award  without  a  3AF0  because  of  the 
narrowness  of  the  criteria  that  must  be  met.  The 
criteria  for  award  without  BAFO,  as  stated  by  the 
respondent,  are: 

1)  full  and  open  competition; 

2)  the  possibility  of  an  award  without 
discussions  must  be  stated  m  the  RFP; 

3)  the  proposal  must  meet  all  the  specific 
requirements  of  the  RFP; 

4)  no  discussions  were  required  or 
conducted ;  and , 

5)  the  contracting  officer  must  certify 
that  the  best  price  available  was 
obtained  by  the  government,  based 
only  on  prior  cost  experience  or 
comparison  to  an  established  price. 
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The  respondent  seated  that  the  criteria,  particularly 
the  first  and  fifth  criteria,  generally  force  the  use 
of  a  BAFO  in  situations  whore  award  without  discussions 
is  felt  to  be  appropriate.  For  example,  when 
competition  is  conducted  after  the  exclusion  of 
sources  le.g.  classified  projects  which  require  special 
access  security  clearances)  an  award  cannot  be  made 
without  discussions  because  of  the  first  criteria.  The 
fifth  criteria  does  not  allow  the  contracting  officer 
to  certify  that  the  best  price  was  obtained  through 
competition;  a  price  history  or  catalog  price  must  be 
available  to  allow  award  without  discussions.  In  the 
past  m  situations  where  award  without  discussions  and 
BAFO  was  felt  to  be  appropriate,  a  BAFO  was  conducted 
because  the  criteria  related  to  price  could  not  be  met. 
The  respondent  suggested  that  the  criteria  be  revised 
in  order  to  expand  the  possibilities  for  award  without 
discussions  and  BAFO. 

The  minor  problems  concern  the  constraints  imposed 
on  the  time  available  for  preparation  of  the  offerors' 
BAFOs ;  the  short  suspenses  can  cause  the  offerors  to 
make  serious  errors  which  cannot  be  corrected.  The 
short-term  solutions  involved  working  closely  with  the 
offerors’  local  offices  and  using  telefax  machines  to 
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minimise  communications  time.  The  proposed  long-term 
solutions  involved  allowing  reasonable  times  for 
responses  and  stressing  the  need  for  full  compliance 
with  RFP  requirements  in  BAFOs . 

Analysis .  The  minor  problems  with  the 
3AF0  process  do  not  represent  serious  flaws  in  the 
process.  By  this  stage,  if  the  process  has  been 
conducted  reasonably,  the  offerors  have  had  adequate 
time  and  information  to  respond  to  a  short  suspense 
request  for  BAFO. 

The  mandate  to  award  without  BAFOs  is,  again,  a 
very  troubling  outgrowth  of  the  streamlining 
initiatives.  Implicit  in  the  mandate  is  the 
willingness  to  accept  proposals  which  do  not  meet  all 
the  minimum  requirements  of  the  RFP.  While  expedient, 
this  willingness  is  clearly  opposed  to  regulatory 
guidance  which  requires  conformance  to  all  aspects  of 
the  RFP.  The  determination  to  award  without 
discussions  and  BAFO  cannot  reasonably  be  made  until 
after  the  proposals  have  been  thoroughly  evaluated  to 
assess  compliance  with  the  RFP  requirements.  The 
criteria  for  award  without  BAFO  should  preclude  such 
situations;  the  third  criterion  requires  compliance 
with  all  RFP  requirements. 
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The  respondent’s  objections  to  the  first  and  fifth 
criteria  appear  to  be  reasonable.  There  does  not 
appear  to  be  any  valid  reason  to  preclude  award  without 
BAFO  when  using  competition  after  exclusion  of  sources. 
Nor  does  there  appear  to  be  a  justification  for 
not  allowing  the  determination  that  the  best  price  has 
been  achieved  to  be  based  on  competition.  One  of  the 
basic  assumptions  of  the  government  acquisition  system 
is  that  competition  results  in  the  best  price 
available;  this  assumption  should  apply  to  source 
selections  as  well  as  sealed  bidding. 

SSAC  Analysis  Report.  (Major  -  1) 

Responses .  This  problem  was  an  unusual 
situation  related  to  the  unauthorised  release  of 
sensitive  information  by  a  member  of  the  SSAC  who  did 
not  work  at  ASD.  This  problem  led  the  respondent  to 
suggest  that  all  outside  participation  in  source 
selections  be  eliminated;  only  ASD  resources  should  be 
utilised . 

Analysis .  The  problem  reported  by  this 
respondent  does  not  appear  to  be  a  common  problem  in 
source  selections.  However,  the  SSAC  Analysis  Report, 
as  defined  by  the  source  selection  regulations,  is  an 
analysis  of  the  SSEB  report,  which  is  a  compilation  of 
the  technical  evaluation  reports  prepared  by  the  area 
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chiefs  based  on  the  evaluation  reports  of  the 
individual  evaluators.  Four  levels  of  technical 
reports  covering  the  same  information  seems  excessive. 

It  would  appear  that  the  SSAC  Analysis  Report- 
represents  an  excessive  requirement  for  documentation. 

SSA  briefing.  Moderate  -  2) 

Responses .  One  respondent  stated  that  too 
many  levels  of  briefings  were  required,  caused  by  the 
involvement  of  different  people  at  each  level.  In  the 
past,  the  respondent  has  complied  viith  the  requirement. 
The  suggested  long-term  solution  was  to  consolidate  or 
eliminate  some  levels  of  briefing. 

The  second  problem  concerned  disagreement  between 
the  SSA  and  the  team  due  to  differences  in  judgment 
between  individuals.  In  addition  to  individual 
differences,  the  respondent  cited  political  naivety  of 
the  team  members  as  a  cause  of  the  problem.  In  this 
case,  the  briefing  was  revised  and  presented  again,  to 
comply  with  the  SSA’s  judgment.  The  respondent 
suggested  that  this  problem  might  be  eliminated  through 
education  in  the  source  selection  process  to  provide  a 
better  foundation  for  making  judgment  calls. 
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Analysis .  The  need  for  numerous  dry  runs  of 
the  SSA  briefing  is  fueled  by  the  visibility  attached 
to  this  activity;  every  level  of  management  wants  to 
ensure  that  the  briefing  is  conducted  successfully. 
This  oversight  is  intended  to  provide  benefit  to  the 
system,  but  is  implemented  in  a  manner  which  imposes 
a  burden.  This  activity  would  seem  to  provide  a  good 
opportunity  for  streamlining  by  reducing  the  number  of 
non-contributing  participants. 

SSA  decision  document.  (Minor  -  2) 

Responses .  Both  minor  problems  involved  the 
need  for  repeated  re-writes  of  the  document  before  the 
technical  justification  was  adequate.  Cited  causes 
were  lack  of  communication  and  lack  of  knowledge  both 
about  the  required  content  and  that  decision  documents 
are  releaseable  under  the  Freedom  of  Information  Act. 
The  problem  was  solved  in  both  cases  through  numerous 
revisions  of  the  document.  Suggested  long-term 
solutions  called  for  involving  technical  personnel  in 
discussions  with  the  SSA  concerning  the  document,  and 
training . 
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Analysis . 


While  the  decision  document 


represents  only  a  minor  problem  in  the  process,  the 
causes  of  this  problem  are  common  to  many  of  the 
serious  problems  which  have  been  identified  in  this 
study.  The  major  influences  involve’  1.1  this  problem 
are  the  lack  of  experience  with  source  Si«* actions  and 
the  failure  of  teamwork.  The  decision  document 
requires  the  input  from  numerous  team  members  to 
adequately  justify  the  SSA's  decision,  but  must  be 
circumspect  in  the  information  revealed.  The  solutions 
suggested  by  the  respondents  appear  to  be  reasonable. 

Contract  Review.  (Major  -  2;  Moderate  -  3) 

Responses .  One  major  problem  involved  the 
need  to  incorporate  numerous  late  changes  while 
accomplishing  the  reviews.  The  problem  was  caused  by 
lack  of  a  coordinated  team  effort,  late  receipt  of 
funds,  and  untimely  receipt  of  technical  changes  from 
the  CR/DR  process.  The  immediate  solution  was  to 
incorporate  the  changes  as  they  were  received,  which 
resulted  in  a  schedule  slip.  The  proposed  long-term 
solution  called  for  better  planning  to  avoid  the  need 
for  numerous  contract  revisions. 

The  second  major  problem  involved  a  contract  which 
required  final  review  and  manual  approval  at  AFSC .  The 
review  notice,  which  was  sent  by  the  ASD  review 
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committee,  was  not  received  by  AFSC.  When  the  document 
was  received  at  AFSC,  it  was  not  on  the  review 
schedule.  The  document  was  held  at  AFSC  for  the  next 
available  reviewer,  resulting  in  a  delay  in  contract 
award.  This  problem  was  caused  by  a  failure  of 
communications  and  lack  of  a  follow-up  on  the  message. 
When  the  problem  occurred,  there  was  no  way  to  correct 
the  situation.  The  suggested  long-term  solution  was  to 
follow  up  on  important  communications  with  other 
organisations . 

Two  of  the  moderate  problems  involved  the 
timeliness  of  contract  reviews.  Manpower  constraints 
and  reviewer  workload  caused  the  inability  to 
accomplish  the  reviews  in  a  timely  manner.  Suggested 
solutions  included  attempting  to  schedule  source 
selections  to  avoid  having  reviews  at  the  end  of  the 
fiscal  year  or  over  holidays,  and  the  establishment  of 
a  control  system  for  reviews  through  time  standards  and 
measurement . 

The  final  situation  involved  proDlems  which  arose 
during  AFSC  review  of  a  contract  on  which  AFSC  had 
waived  the  RFP  review.  The  problem  was  caused  by  the 
AFSC  policy  of  waiving  RFP  reviews  when  their  review 
workload  is  heavy.  The  immediate  solution  included 
obtaining  a  'clarification'  from  the  contractor,  and 
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agreeing  to  include  AFSC's  required  changes  in  the 
first  modification  to  the  contract.  The  suggested 
long-term  solution  required  awareness  at  AFSC  of  the 
critical  nature  of  ?.F?  reviews. 

Analysis .  Five  of  the  twelve  respondents 
rated  contract  review  as  a  significant  problem.  The 
problems  can  generally  be  attributed  to  lack  of 
teamwork,  failure  of  communications,  and  manpower 
constraints.  That  AFSC,  the  originator  of  the 
streamlining  initiatives,  would  raise  obstacles  to 
award  due  to  their  failure  to  review  the  RFP  is  a 
troubling  aspect  of  this  problem. 

Contract  execution.  (Major  -  1) 

Responses .  This  problem  involved  a  source 
selection  which  was  set  aside  for  small  business.  Prior 
to  award  of  a  competitive  small  business  set-aside,  the 
unsuccessful  offerors  must  be  notified  of  the  intended 
awardee.  This  requirement  exists  to  allow  the 
unsuccessful  offerors  to  challenge  the  small  business 
status  claimed  by  the  apparent  successful  offeror.  In 
this  case,  the  notification  prompted  an  unsuccessful 
offeror  to  submit  a  protest  before  award  on  an  issue 
unrelated  to  business  size.  The  problem  was  caused  by 
the  requirement  for  a  pre-award  notice  to  the 
unsuccessful  offerors  (post  award  notices  are  required 


in  other  circumstances) .  The  contract  award  was 
delayed  while  the  protest  was  processed  as  required  by 
regulations.  The  respondent  suggested  that  either  the 
requirement  for  pre-award  notice  be  eliminated,  or  that 
notification  of  all  participants  in  the  competition  bo 
given  earlier  in  the  process.  This  would  allow  any 
challenges  to  be  resolved  without  affecting  the 
contracting  schedule. 

Analysis .  The  experience  of  the  researcher 
and  responses  of  the  interview  subjects  indicate  that 
this  problem  is  not  a  common  one,  due  to  the  small 
number  of  small  business  set-asides  awarded  through 
source  selections.  However,  the  problem  is  a 
significant  one  when  it  occurs.  The  requirement  for 
pre-award  notice  to  unsuccessful  offerors  seems  to  be 
an  ill-conceived  requirement  which  should  be  changed  to 
eliminate  the  invitation  it  offers  for  protests  before 
award.  This  change  would  require  a  modification  to 
the  FAR. 

Contract  distribution.  (Minor  -  1) 

Responses .  One  respondent  cited  printing 
leadtime  as  a  minor  problem  with  this  activity,  caused 
by  the  workload  in  the  printing  office.  The  respondent 
has  dealt  with  this  problem  by  requesting  priority 
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printing.  Tho  suggested  solution  was  to  provide 
greater  printing  capabilities. 

Analysis .  This  represents  a  minor  problem 
which  should  bo  correctable  through  increased 
manpower  or  additional  contracting-out  of  printing 
services . 

Cost/Bonefit  of  Source  Selection  Activities 

In  this  portion  of  the  interview,  the  subjects 
were  asked  to  review  rhe  source  selection  activities 
to  identify  those  which  require  a  greater  expenditure 
of  effort  or  resources  than  the  benefit  derived.  For 
each  activity  identified,  the  respondent  was  asked  to 
suggest  a  more  cost-beneficial  method  of  completing  or 
achieving  the  objective  of  the  questioned  activity. 

The  respondents  identified  15  activities  which 
they  felt  could  be  performed  more  cost  effectively  or 
deleted  without  detriment  to  the  source  selection 
process.  These  responses  are  shown  in  Appendix  C. 

No  activities  received  unanimous  identification  as 
requiring  a  greater  expenditure  of  effort  or  resources 
than  the  benefit;  received.  Many  of  the  comments  made 
during  this  portion  of  the  interview  echoed  comments 
from  the  first  section. 
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The  significant  suggestions  which  have  not  bean 
addressed  include: 

1)  limiting  the  number  of  participants 
in  'ie  ASP  by  using  fewer  people 
with  broader  backgrounds; 

2)  deleting  the  requirement  for 
conducting  a  market  survey  for 
identification  of  sources; 

3)  deleting  the  requirement  for  AFSC 
review  of  RFPs; 

4)  combining  or  reducing  the  number 
of  technical  evaluation  reports 
to  reduce  the  time  and  redundancy 
involved;  and, 

5)  U3ing  more  oral  discussions  and  fewer 
CRs  and  DRs . 

With  the  exception  of  deleting  AFSC  review  of  RFPs, 
which  can  cause  problems  at  final  contract  review, 
these  suggestions  appear  to  be  reasonable.  Deletion  of 
the  market  survey  would  require  a  change  to  the  FAR;  a 
change  to  the  Competition  in  Contracting  Act  might  also 
be  required.  Changing  the  technical  report 
requirements  would  require  modifications  to  AFR  70-15 
and  AFR  70-30.  The  other  suggestions  could  be 
implemented  by  changing  either  the  ASD  Supplement  to 
the  source  selection  regulations  or  through  local 
policy  letters. 
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Local 


Ini tiati vos 


This  section  of  the  interview  was  intended  to 
determine  whether  any  ASD  contracting  organisations  had 
implemented  unique  local  initiatives  to  accelerate 
source  selection  schedules.  Subjects  were  asked  to 
identify  any  unusual  moans  which  had  been  used  to  avoid 
problems  or  accelerate  schedules  for  source  selections, 
describe  how  effective  those  actions  had  been,  and 
state  whether  those  methods  would  be  used  again  in 
similar  circumstance* 

It  appears  that  very  lew  initiatives  to  accelerate 
source  selection  schedules  have  been  instituted  by 
individual  contracting  directorates  at  ASD.  The 
initiatives  named  by  the  respondents  included: 

1)  establishing  detailed  milestone 
schedules  for  each  team; 

2)  issuing  CRs  and  DRs  in  one  batch; 

3)  issuing  modifications  after 
contract  award  to  avoid  the  need 
for  discussions; 

4)  obtaining  minimal  cost  data  and 
relying  on  competition  for  price 
justification; 

5)  accomplishing  the  Mid-Term 
briefing  through  a  memorandum; 

6)  using  several  Solicitations 
for  Information  and  holding 
individual  meetings  with 
potential  offerors  to  discuss 
requirements  definition; 
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7) 


asking  offerors  to  provide 
information  about  rent-free  use 
of  government-owned  facilities  or 
property  and  about  EEO  clearances 
15  days  after  Issuance  of  the  RFP, 
rather  than  waiting  for  the 
information  to  be  submitted  with 
the  proposals;  and, 

8)  providing  written  instructions 
to  the  technical  team  on  CR/DR 
preparation  and  processing. 

The  respondents  felt  that  these  actions  were  effective, 
and  should  be  used  again. 

The  third  initiative,  modifying  the  contract  after 
award  to  avoid  the  need  for  discussions,  is  viewed  by 
the  researcher  as  a  questionable  tactic;  the  objections 
to  this  technique  were  discussed  in  an  earlier  section 
of  this  chapter. 

The  fourth  initiative,  obtaining  minimal  cost  data 
and  limiting  cost/price  analysis,  connot  really  be 
considered  a  local  initiative;  recent  DoD  policy 
interpretations  have  advocated  this  approach  for 
competitive  acquisitions.  This  technique  has  not  been 
widely  used,  however.  The  respondent's  recommendation 
of  the  technique  demonstrates  that  it  is  applicable  to 
major  source  selections. 
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The  sixth  initiative,  soliciting  information  from 
industry  for  use  in  the  definition  of  requirements,  is 
a  viable  approach  to  ensuring  that  requirements  are 
realistic  and  achievable.  Care  must  be  exercised, 
however,  in  the  conduct  of  private  meetings  with 
potential  offerors  to  ensure  that  requirements  are  not 
tailored  to  favor  a  particular  approach  and  that  all 
interested  sources  receive  equal  access  to  the 
government  and  ar*>  treated  fairly. 

The  seventh  initiative,  obtaining  early 
notification  about  rent-free  use  of  government-owned 
assets  and  about  needed  EEO  clearances,  is  an  excellent 
way  to  ensure  that  the  necessary  approvals  and 
clearances  are  received  in  a  timely  manner.  Obtaining 
these  approvals  and  clearances  can  be  very  time- 
consuming;  early  notification  provides  an  extra  cushion 
to  ensure  that  the  award  is  not  delayed  by  these 
administrative  actions. 

The  other  initiatives  might  provide  some 
administrative  benefit,  but  do  not  represent 
significant  improvements  or  changes  to  the  process. 

With  the  exception  of  the  third  initiative,  these 
techniques  could  be  employed  without  a ”  j  changes  to 
existing  regulations  or  policy  documents. 
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General  Comments  about  Source  Selection 

The  final  portion  of  the  interview  was  intended  as 
an  open  forum  to  allow  the  subjects  to  make  comments  on 
any  aspect  of  source  selection.  Th^  comments  all 
related  to  two  issues:  1)  von  and 

experience  of  personnel,  *  c'j  of  the 

streamlining  initiative*  of  the 

comments  will  be  divided  in  .  ,nner.  For  each 

issue,  the  discussion  will  begin  with  the  comments  made 
by  the  respondents;  an  analysis  of  the  comments  will 
follow. 


Lack  of 


lerlence  and  education  of  personnel. 


Comments .  Four  comments  were  made  about  the 


lack  of  experience  and  education  of  personnel  involved 
in  source  selections. 

One  subject  stated  that  people  are  very  poorly 
prepared  for  source  selections.  The  lack  of 
preparation,  combined  with  tight  schedules,  leads  to 
frustration  on  both  the  government  and  offeror  teams, 
and  increases  the  probability  that  major  mistakes  will 
be  made.  The  respondent  suggested  that  extra  guidance 
be  provided  for  inexperienced  program  managers,  which 
the  respondent  felt  was  the  worst  place  for 
inexperience  on  a  source  selection  team. 
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A  second  respondent  suggested  that  contracting 
teams  should  always  pair  an  experienced  person  with  a 
trainee.  This  would  ensure  a  certain  amount  of 
experience  on  every  source  selection  team,  rather  than 
continuously  cycling  inexperienced  people  through 
source  selections  as  square- f i 1 1  mg  exercises  for 
promotions . 

The  two  remaining  comments  both  cited  the  lack  of 
sustained  learning;  high  turnover  causes  each  source 
selection  to  be  a  learning  experience.  One  of  these 
respondents  pointed  out  that  the  formal  education 
required  for  contracting  personnel  ignores  source 
selection.  Both  suggested  that  ASD/PMPS  be  expanded  to 
include  experts  to  take  the  lead  in  writing  source 
selection  documents  and  to  serve  as  the  core  of  source 
selection  teams,  to  be  augmented  by  program  office 
personnel  who  would  maintain  the  program  continuity 
after  completion  of  the  source  selections. 

Analysis .  The  suggestions  offered  in  this 
portion  of  the  interview  to  overcome  the  lack  of 
education  and  experience  are  all  valuable  suggestions. 

The  most  effective  solution  would  establish  a  pool 
of  experts  in  a  centralized  office  to  run  all  the 
source  selections  at  ASD.  This  would  ensure  an 
experienced  core  to  direct  each  source  selection,  while 
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braining  the  inexperienced  personnel  drawn  from  the 
program  office  to  augment  the  teams  and  provide  program 
continuity.  The  fiscal  practicality  of  this  suggestion 
cannot  be  accurately  assessed  by  the  researcher,  but  it 
appears  to  be  a  more  practical  solution  than 
establishing  dedicated  source  selection  teams  m  each 
Deputy  or  conducting  every  source  selection  with  an 
inexperienced  team. 

Effects  of  streamlining  initiatives. 

Comments .  Four  respondents  also  made 
comments  which  related  to  the  recent  streamlining 
initiatives;  all  were  dissatisfied  with  the  effects 
which  the  initiatives  have  had  on  the  source  selection 
process . 

One  respondent  stated  that  the  current  policy 
stresses  schedule  above  all  other  considerations.  The 
respondent  felt  that  quality  should  be  the  first 
priority,  to  ensure  that  good  decisions  are  made  and 
that  sole  source  engineering  changes  after  contract 
award  are  minimized. 

A  second  respondent  stated  that  source  selections 
should  be  allowed  to  ‘run  their  natural  course'  rather 
than  being  forced  into  artificial  time  constraints 
which  ignore  the  complexity  or  unusual  aspects  of  some 
acquis  1 tions . 
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Another  respondent  took  exception  to  the  mandated 
use  of  award  without  discussions.  The  respondent 
stated  that,  while  discussions  may  have  been  used  too 
often  in  the  past,  a  realistic  approach  would  accept 
that  acquisitions  of  complex  systems  usually  require 
discussions . 

The  last  comment  emphasized  the  added  requirements 
that  have  been  levied  on  the  system  by  ASD  in  the  name 
of  streamlining.  The  Source  Selection  Management  Group 
(SSMG)  is  a  new  bureaucracy  which  has  been  created, 
outside  of  the  tracked  time,  to  review  the  source 
selection  planning.  The  Acquisition  Review  Team  CART) 
replaced  the  old  ‘murder  board*  which  reviewed  RF?s . 
While  the  murder  board  was  a  single  meeting,  the  ART  is 
a  three  phase  process,  also  outside  of  the  tracked 
portion  of  the  process: 

1)  the  ART  may  be  convened  before 
the  ASP  to  participate  in 
development  of  the  acquisition 
strategy ; 

2)  the  ART  must  convene  to  conduct 
a  detailed  review  of  the  draft 
RF?  and  all  aspects  of  the 
acquisition;  and, 

3)  after  comments  are  received  from 
industry  reviews  of  the  draft 
P.FP ,  the  source  selection  team's 
suggested  disposition  of  the 
comments  is  reviewed  by  the  ART 
chairman.  The  chairman  has  the 
discretion  to  re-convene  the  ART 
before  issuance  of  the  RFP. 
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The  reviews  conducted  by  the  ART  have  made  other 
established  review  requirements  redundant: 

1)  an  engineering  review  of  the 
technical  aspects  is  required 
before  the  ART.  This  review  is 
repeated  during  the  ART; 

2)  ASD/PMC  and  AFCLC/JAN  conduct 
compliance  and  legal  reviews 

as  part  of  the  ART.  These  reviews 
are  repeated  after  the  ART  comments 
are  addressed;  and, 

3)  the  ART  reviews  the  source 
selection  planning,  which  has 
alreadly  be^n  reviewed  by  the 
SSMG. 

Additional  requirements  such  as  these  have  added  to  the 
total  time  required  to  complete  a  source  selection, 
without  affecting  the  portion  of  the  process  which  is 
tracked  against  the  command  goal,  which  represents  only 
a  very  small  part  of  the  process. 

Analysis .  The  comments  on  the  effects  of 
streamlining  reflect  a  great  deal  of  cynicism  toward 
the  initiatives.  The  streamlining  initiatives  were 
intended  to  lessen  the  time  required  to  conduct  source 
selections.  The  impementation  of  these  initiatives 
brought  some  new  requirements,  but  no  existing 
requirements  were  removed.  The  oversight  mandated  by 
the  initiatives  is  applied  to  a  very  limited  segment  of 
the  process,  ignoring  any  schedule  growths  which  occur 
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outside  of  that  segment  of  the  process.  The  segment 
which  is  covered  by  the  time  constraint  i3  perhaps  the 
most  disciplined  and  most  critical  portion  of  the 
process,  benefiting  least  from  the  imposition  of  time 
limits . 

Additionally,  there  is  presently  great  emphasis  on 
Total  Quality  Management  (TQM)  at  ASD,  but  no  mention 
of  the  application  of  TQM  to  the  source  selection  process, 
indeed,  the  respondents  felt  that  the  opposite  applies 
to  source  selection  because  the  emphasis  is  on  schedule 
over  quality  or  other  considerations . 

Chapter  Summary 

Thi3  chapter  described  the  subjects  who  took 
part  in  the  interviews  for  this  study,  presented  the 
findings  obtained  from  those  interviews,  and  analysed 
those  findings. 

The  interviews  revealed  that  participants  in 
recent  source  selections  at  ASD  perceive  problems  in 
the  source  selection  process.  The  most  significant 
problems  identified  by  the  interview  subjects  were  in 
the  following  areas: 

1)  preparation  of  source  selection 
documents  such  as  the  evaluation 
criteria,  evaluation  standards, 
and  the  instructions  to  offerors; 
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2)  preparation  of  the  purchase 
request  package  which  defines  the 
requirements  of  the  acquisition; 

3)  review  of  the  Request  for  Proposal; 
and , 

4)  the  technical  evaluation. 

The  cited  causes  of  these  problems  were: 

1)  lack  of  understanding  of  the  source 
selection  process; 

2)  lack  of  experience  in  source  selection; 

3)  lack  cf  training  of  the  participants; 

4)  lack  of  quality  control  on  technical 
documents ; 

5)  excessive  oversight  requirements;  and, 

6)  effects  of  streamlining  initiatives. 

^he  conclusions  and  recommendations  resulting  from 
the  research  will  be  addressed  m  thv.  following 
chapter . 


104 


Conclusions  and  Recommendations 

Overview 

This  chapter  draws  conclusions  from  the  findings 
presented  in  Chapter  IV  to  answer  the  investigative 
questions  posed  by  this  study.  After  the  discussion  of 
the  investigative  questions,  some  general  conclusions 
will  be  presented.  These  conclusions  will  be  followed 
by  recommendations  related  to  the  source  selection 
process,  limitations  of  this  study,  and  recommendations 
for  future  research. 

Conclusions 

The  interviews  conducted  to  gather  data  for  this 
study  revealed  that  contracting  personnel  involved  in 
recent  source  selections  at  Aeronautical  Systems 
Division  (ASD)  perceive  problems  in  the  conduct  of 
source  selections.  The  problems  identified  by  this 
study  do  not  indicate  fundamental  flaws  in  the  basic 
source  selection  process;  the  problems  indicate  the 
need  for  educational,  procedural,  and  organisational 
climate  changes,  and  modification  or  elimination  of 
individual  tasks  within  the  process. 
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Investigative  Question  I:  What  are  the  major 


factors  contributing  to  the  length  of  the  process?  Can 
these  factors  be  controlled? 

This  pilot  study  has  identified  several  factors 
which  appear  to  be  significant  contributors  to  the 
length  of  the  source  selection  process.  All  of  the 
problems  identified  by  this  study  can  be  attributed  to 
one  or  more  of  these  factors,  each  of  which  will  be 
briefly  addressed  following  the  list.  These  factors 
are : 

1)  lack  of  understanding, 
experience,  and  training; 

2)  lack  of  teamwork; 

31  lack  of  quality  control; 

41  lack  of  written  guidance; 

51  effects  of  ASD  streamlining 

initiatives; 

6)  excessive  oversight; 

71  unneccessary  or  excessive 
requirements ; 

81  manpower  constraints;  and, 

91  politics. 

Lack  of  understanding ,  experience,  and  training 
was  the  most  frequently  cited  cause  for  problems 
experienced  by  the  respondents.  The  respondents  felt 
that  most  participants  in  source  selections  are  ill- 
prepared,  lacking  both  a  general  understanding  of  the 
process  and  a  knowledge  of  their  individual  tasks  and 
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responsibilities.  The  absence  of  formal  training  in 
source  selection  requires  the  participants  to  learn 
on  the  job,  often  having  to  repeat  tasks  several  times 
to  accomplish  them  successfully. 

The  failure  of  teams  to  work  together  effec.  vely 
can  cause  problems  m  planning  the  sourco  selection, 
resolving  problems  which  arise  during  its  conduct,  and 
accomplishing  multi-disciplinary  tasks  efficiently. 

This  lack  of  teamwork  can  be  attributed  to  conflict? 
resulting  from  the  different  roles  served  by  the 
various  functionals  on  the  team  and  the  lack  of 
understanding  of  the  value,  purpose,  and  requirements 
of  those  roles.  Those  conflicts  can  be  seen  between 
program  managers  and  contracting  personnel,  between 
program  control  and  contracting,  and  betwo  ..  users  and 
the  acquisition  community. 

Many  problems  encountered  by  the  respondents 
wore  caused  by  inadequate  quality  control  of  source 
selection  and  technical  requirements  documents.  The 
most  significant  problems  caused  by  failure  of  adequate 
quality  control  involved  deficiencies  m  and  lack  of 
correlation  between  the  evaluation  criteria,  evaluation 
standards,  and  instructions  to  offerors;  defective  PR 
packages;  and  inadequate  technical  evaluation 
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documentation.  The  documents  required  repeated 
revisions,  and  in  some  cases  led  to  the  need  for 
modifications  to  the  solicitations  and  proposals  and 
to  ro-accompl ishment  of  the  technical  evaluations. 

The  lack  of  written  guidance  was  a  significant 
factor  in  the  problems  encountered  m  ?R  package 
preparation,  wher«  respondents  cited  the  absence  of  a 
standard  defining  the  contents  of  the  package.  The 
respondents  also  cited  the  lack  of  appropriate  'sample' 
documents  as  a  factor  in  technical  evaluation  and 
performance  risk  analysis  documentation  problems.  The 
lack  of  written  guidance  resulted  in  incomplete  ?R 
packages  and  inadequate  technical  evaluation 
documentation  which  had  to  be  returned  to  the 
originators  for  significant  revisions,  adding  to  the 
length  of  the  source  selection. 

Several  respondents  cited  the  additional 
requirements  implemented  by  the  ASD  streaml lnlng 
initiatives,  such  as  the  Source  Selection  Management 
Group  and  the  Acquisition  Revi ew  Team  (ART) ,  as 
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contributors  to  the  length  of  the  process.  The 
respondents  stated  that  these  requirements  added 
significantly  to  the  total  time  required  to  complete  a 
source  selection  by  increasing  the  up-front  time  which 
is  not  monitored  against  the  command  goal  of  120  days 
from  RFP  release  to  SSA  decision. 

Excessive  management  oversight  was  cited  as  a 
significant  problem  by  many  of  the  interview  subjects. 
Tho  excessive  oversight  was  most  evident  .  n  the 
Business  Strategy  Panel  (BSPJ /Acquisi tion  Strategy 
Panel  (ASP) ,  where  the  Judgment  of  those  most 
knowledgeable  about  the  program  was  frequently 
overturned;  the  RFP  and  contract  review  cycles,  where 
the  respondents  encountered  numerous  redundant  reviews; 
the  requirement  for  Source  Selection  Authority  tSSA) 
approval  of  tho  Clarification  Requests  (CRs)  and 
Deficiency  Reports  (DRs) ,  which  delayed  the  issuance  of 
CRs  and  DRs  and  sometimes  limited  the  resolution  of 
source  selection  issues;  and  in  the  final  SSA  briefing, 
which  required  several  levels  of  briefing  because  of 
the  numerous  individuals  involved. 

The  respondents  cited  several  unnecessary  or 
excessive  requirements  which  add  to  the  length  of  the 
process.  The  excessive  requirements  included  numerous 
redundant  management  and  RFP  reviews,  duplication  in 
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the  marked  survey  and  synopsis  requirements,  the  need 
to  send  RFP3  to  all  respondents  to  the  synopsis, 
extensive  cost  and  price  analysis,  and  the  F.SAC 
Analysis  Report.  Unnecessary  requirements  included  the 
Quick  Look  briefing,  Most  Probable  Cost  estimate,  and 
Mid-Term  briefing. 

Manpower  constraints  were  cited  as  causes  for 
delays  in  contract  preparation  and  timely  completion  of 
SFP  and  contract  reviews.  The  respondents  felt  that 
clerical  support  to  source  selections  was  frequently 
inadequate,  and  that  the  workloads  of  review  personnel 
caused  delays  in  reviews  or  waiver  of  AFSC  RrP  reviews. 

Politics  was  cited  as  a  problem  in  the  ASP, 
requirements  definition  process,  and  source  selection 
briefings.  The  respondents  felt  that  politics 
frequently  interfered  with  the  exercise  of  sound 
business  judgment. 

All  of  these  factors  can  be  controlled  to  some 
extent.  Most  of  the  factors  can  be  influenced  locally, 
with  the  factors  involving  training  and  experience, 
teamwork,  quality  control,  written  guidance,  and  ASD 
streamlining  initiatives  being  totally  under  local 
control.  Some  aspects  of  the  factors  involving 
oversight  and  unnecessary/excessive  requirements 
factors  are  controlled  by  AFSC  or  higher 
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authorities.  The  last  factor,  politics,  can  be 
Influenced  to  a  limited  extent  at  all  levels  in  the 
bureaucracy,  but  cannot  be  eliminated  from  the 
acquisition  process. 

The  changes  needed  to  mitigate  the  Influences  of 
these  factors  will  be  addressed  later  in  this  chapter. 

Investigative  Question  3:  What  local  initiatives 
have  been  implemented  to  expedite  source  selections? 

Kow  effective  have  thoso  initiatives  been? 

This  study  identified  no  significant  local 
initiatives.  The  respondents  identified  some  actions 
which  have  been  taken  to  expedite  source  selections, 
but  all  have  been  very  limited  in  scope  and  effect. 

This  lack  of  local  initiatives  is  troubling  m 
that  the  participants  all  were  able  to  cite  suggested 
solutions  to  the  problems  which  are  encou1  ered  in 
source  selection,  but  had  taken  no  action  to  have  these 
solutions  implemented.  It  is  possible  that  a  cause  for 
this  lack  of  initiative  by  the  contracting  personnel 
can  be  seen  in  the  reaction  to  attempts  which  the 
respondents  had  made  to  influence  decisions  which  they 
felt  would  be  detrimental  to  their  programs.  One 
respondent  had  attempted  to  influence  ASP  guidance; 
another  had  attempted  to  influence  an  SSA  mandate  that 
a  source  selection  would  be  conducted  without 
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discussions .  Both  respondents  ware  unsuccassful  m 
their  attempts  to  reverse  decisions  which  thay  fait 
were  harmful  to  thair  programs.  Thasa  failad  attempts 
may  have  lad  to  a  feeling  of  powerlessness  in  tha  faca 
of  tha  buraaucracy  which  controls  tha  procass. 

General .  Tha  general  araas  which  must  ba  improvad 
ara  shown  balow,  rankad  by  tha  rasaarchar  in  descending 
ordar  of  importanca: 

1)  skill  and  knowledge  of  participants; 

2)  datarmination  of  appropriate  lavals 
of  management  oversight; 

3)  alimination  of  non-value-added 
activi tias ; 

4)  quality  of  businass  dacisions;  and, 

5)  quality  of  sourca  salaction  and 
tachnical  raquiramants  documants. 

Thasa  araas  for  improvamant  ancompass  problams 

influancad  by  all  tha  factors  dascribad  abova. 

Improvamants  in  the3a  araas  would  Itssic  xiitarr.e 

selection  activities  to  those  essential  to  the 

integrity  of  the  procass,  remove  extraneous 

participants  from  the  procass,  and  attempt  to  ensure 

that  actions  are  completed  correctly  the  first  time,  in 

a  manner  that  serves  the  best  interests  of  the  program 

and  the  government.  The  actions  recommended  to  bring 

about  the  needed  improvements  are  discussed  below. 
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Recommendations 


The  recommendations  of  this  study  can  be  divided 
into  two  categories:  specific  recommendations  related 
to  the  individual  activities  of  the  source  selection 
process  and  general  recommendations  related  to 
improvements  in  performance  of  the  process. 

Specific  Recommendations  for  Individual 
Activities.  The  specific  recommendations  for  source 
selection  activities  were  addressed  in  detail  in 
Chapter  IV.  These  recommendations  are  also  listed  in 
Appendix  D  for  the  reader's  convenience.  A  summary  of 
the  most  significant  recommendations,  which  will  be 
followed  by  rationale  for  the  suggested  changes,  is: 

1)  changing  public  law/FAR  requirements 
to  allow  screening  of  CICA  synposis 
respondents  and  to  charge  consultants 
and  marketeers  for  RFPs  as  a  cost  of 
doing  business; 

2)  developing  instructions  to  define 
PR  package  requirements; 

3)  establishing  the  Acquisition  Review 
Team  (ART!  as  the  sole  RFP  review 
at  ASD; 

4)  completing  the  initial  technical 
evaluation  before  determining 
whether  to  award  without  discussions; 

5J  severely  limiting  cost  analysis; 

6)  deleting  the  requirement  for  Most 
Probable  Cost  (MPC)  estimates; 

7)  deleting  the  SSAC  Analysis  Report; 

8)  reducing  the  participants  m  the 
S3A  briefing;  and, 

9)  raising  manual  approval  threshholds. 
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The  regulations  currently  require  the  government 
to  expend  considerable  resources  to  provide  RFPs ,  at  no 
charge,  to  all  respondents  to  the  CICA  synopsis. 
Frequently,  many  of  the  respondents  are  consultants  or 
marketeers  who  are  not  potential  sources  for  the  goods 
or  services  being  acquired.  These  non-manufacturing 
firms  then  use  the  information  provided  at  government 
expense  to  solicit  business  from  potential  offerors. 

Tho  regulations  should  be  changed  to  allow  the 
government  to  charge  these  firms  a  reasonable  foe  for 
the  reproduction  of  the  RFPs. 

3ecause  some  ASD  organisations  do  not  have 
standards  to  define  PR  package  requirements,  the 
packages  delivered  to  dontractmg  offices  are 
frequently  incomplete  or  otherwise  inadequate. 

These  packages  are  reviewed  i nd  returned  to  the  program 
offices  for  correction,  resulting  in  the  loss  of 
considerable  time.  These  problems  with  PR  package 
requirements  could  be  reduced  by  the  development  of 
operating  instructions  defining  the  content  and  format 
of  PR  package  documents. 


114 


Tho  currant  RF?  reviow  cyclo  for  a  high  dollar 
program  includes  tho  following  sequential  reviews: 

1 )  ART , 

2)  JAG, 

3)  directorate  contract  review 
committee , 

4)  ASD  contract  review  committee,  and, 

5)  AFSC  contract  review  committee. 

Representatives  from  JAG  and  the  directorate  and  ASD 
review  committees  participate  m  the  ART;  each  conduct 
another  review  after  the  ART  is  completed.  These 
redundant  reviews  could  be  avoided  by  having  all 
participants  approve  the  RFP  upon  completion  of  the 
ART. 

In  at  least  one  interview  for  this  study,  it  was 
reported  that  the  SSA  had  mandated  award  without 
discussions  at  the  start  of  the  source  selection.  The 
determination  to  award  without  discussions  should  be 
based  on  the  adequacy  of  the  proposals  received;  all 
requirements  of  the  RFP  must  be  satisfied  before 
contract  award.  A  mardated  award  without  discussions 
places  the  highest  priority  on  schedule,  and  could 
compromise  the  quality  or  increase  the  cost  o:  a  system 
by  allowing  contract  award  against  a  deficient 
proposal.  To  ensure  that  all  requirements  of  the  RFP 
are  met,  the  decision  to  award  without  discussions 
should  not  be  made  before  completion  of  the  initial 
technical  evaluation. 
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In  a  source  selection,  the  contract  price  Is  not 
reached  through  negotiation;  rather,  the  proposed  price 
of  the  proposal  which  is  determined  to  represent  the 
best  value  is  accepted  by  the  government  in  a 
competitive  environment.  Detailed  cost  analysis 
is  of  very  little  value,  because  the  government  cannot 
enforce  any  of  the  findings;  only  general  areas  of 
disagreement  can  be  discussed.  Therefore,  the 
considerable  effort  involved  in  the  routine  performance 
of  audits  and  cost  analysis  is  wasted.  When  prices  are 
determined  through  competition,  cost  analysis  should 
be  limited  to  the  minimum  required  to  ensure  that  the 
cost  proposal  includes  all  requirements  of  the  RFP. 

The  rationale  for  deletion  of  the  MPC  is  also 
based  on  the  method  of  determining  prices  in 
competitive  environments.  The  MPC  is  a  time-consuming 
effort  which  produces  no  benefit  to  the  Air  Force  in  a 
competitive  acquisition  and  should,  therefore,  bo 
eliminated. 

The  SSAC  report  is  an  analysis  of  the  SSEB  report, 
which  is  a  compilation  of  the  technical  reports 
prepared  by  the  area  chiefs  from  the  reports  prepared 
by  the  individual  evaluators.  Four  levels  of  written 
reporting  of  the  same  information  appears  to  be 
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excessive.  Any  information  not  included  m  the  SSEB 
report  which  the  SSAC  wishes  to  convey  to  the  SSA  could 
be  presented  at  the  final  briefing  without  the  need  for 
an  additional  written  report. 

The  last  activity-specific  recommendation  cited 
above  involves  increases  in  the  manual  approval 
threshholds,  which  would  reduce  the  number  of  contracts 
requiring  final  reviews  and  manual  approval  above  the 
contacting  directorate  level.  The  A5D  review  committee 
and  AFSC  would  still  have  the  opportunity  to  review  the 
RFPs  for  these  acquisitions,  which  should  be  adequate 
because  the  RFPs  are  essentially  model  contracts. 

General  Recommendations.  The  first  general 
recommendation  involves  training.  There  is  a  senou3 
need  for  development  of  an  extensive  training  program 
to  ensure  that  source  selection  participants  thoroughly 
understand  the  process  and  their  tasks  within  that 
process.  Both  multi-disciplinary  and  specialised  task- 
or  discipline-oriented  learning  modules  are  needed. 

One  possible  approach  would  begin  with  a  lecture  and 
discussion  session  for  all  participants,  to  provide  an 
overview  of  the  process  and  make  clear  the 
relationships  between  the  various  activities  and 
participants  in  the  process.  This  introductory  session 
could  be  followed  by  smaller  ’workshop"  sessions  which 
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w-»uld  include  instruction*!  sessions  on  specific  task*- 
an4  multi-discipl inary  sessions  to  include  activities 
eh  >s  case  studies  and  exercises  in  planning, 
strat:*$y  formulation,  problem-solving,  and  document 
pv*' •pt',rati»>n  and  *eview.  In  addition  to  providing 
pract  cal  experience  with  skills  required  in  source 
selection,  these  multi-disciplinary  exercises  would 
foster  the  teamwork  which  is  essential  for  effective 
conduct  of  the  process. 

The  second  general  recommendation  calls  for 
application  of  Total  Quality  Management  (TQM)  to  the 
source  selection  process.  The  problems  revealed  by 
this  study  do  not  appear  to  be  systemic  in  nature; 
rather  they  oftan  resulted  from  the  inclusion  of 
non-value-added  activities  or  procedures,  failures 
of  quality  control,  or  implementation  of  ill-conceived 
decisions.  The  process,  including  AFSC  activities  and 
participation,  and  the  ASD  streamlining  initiatives 
should  be  subjected  to  a  complete  TQM  review  to  ensure 
that  all  decisions  and  activities  provide  a  benefit  to 
the  process. 

The  third  recommendation  involves  the  possibility 
of  establishing  a  centralized  source  selection  office 
to  provide  core  teams  of  experienced  personnel  to 
conduct  maj'or  source  selections  for  the  product 
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division,  augmented  by  personnel  from  the  program 
offices.  The  practicality  of  establishing  such  an 
organisation  should  be  assessed,  because  it  would 
appear  that  maintaining  such  expertise  and  applying  it 
to  every  source  selection  would  provide  significant 
benefit  to  the  process. 

Llml tations 

The  validity  and  applicability  of  this  research  is 
subject  to  several  limitations. 

The  research  was  conducted  solely  at  ASD  due  to 
constraints  on  time  and  resources.  Much  of  the 
research  might  be  applicable  to  other  AFSC  product 
divisions,  but  some  findings  are  unique  to  ASD.  It  is 
not  known  whether  any  of  the  research  is  applicable  to 
other  commands  of  the  Air  Force  or  to  the  other  Armed 
Services . 

The  research  was  conducted  with  a  small  sample; 
only  12  individuals  were  interviewed.  However,  the 
population  from  which  the  sample  was  drawn  consisted  of 
ASD  contracting  personnel  who  had  completed  major 
source  selections  between  February  1988  and  June  1989; 
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only  13  individuals  made  up  the  entire  population.  The 
twelve  individuals  who  comprised  the  sample  wore 
sufficiently  experienced  and  knowledgeable  in 
government  contracting  and  honest  and  forthcoming 
during  the  Interviews  to  be  considered  a  credible 
sample  for  a  pilot  research  study. 

The  research  was  limited  to  the  perspectives  of 
contracting  personnel.  While  these  Individuals  had 
valuable  insight  into  the  entire  source  selection 
process,  the  views  expressed  may  have  been  affected  by 
parochial  interests. 

The  information  gathered  during  the  research 
effort  was  qualitative  in  nature,  not  quantitative,  and 
is  therefore  subject  to  interpretation.  However, 
because  this  was  a  pilot  study,  the  desired  information 
could  be  elicited  only  through  narrative  expressions  of 
the  participants'  experiences. 
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Recommendations  for  Further  Research 


Repi lcation  of  this  study  at  other  AFSC  product 
divisions,  other  Air  Force  commands,  and  buying 
commands  of  other  Armed  Services  is  recommended.  This 
replication  would  determine  whether  the  same  problems 
are  experienced  throughout  government  source  selections 
and  could  result  in  additional  recommendations  for 
improving  the  process. 

Similar  studios  could  also  be  conducted  with 
personnel  from  other  disciplines,  such  as  program 
management  and  engineering.  While  contracting 
personnel  have  an  insight  into  the  entire  process,  the 
views  of  source  selection  participants  from  other 
functional  areas  would  provide  a  broader  perspective. 

Additional  research  is  recommended  into  the 
individual  activities  identified  by  this  study  as 
significant  problems  in  the  source  selection  process. 
Suggested  activities  for  this  research  include 
preparation  of  evaluation  criteria  and  standards, 
Instructions  to  offerors,  PR  package  development,  SF? 
preparation  and  review,  and  technical  evaluation. 
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Another  worthwhile  research  effort  would  be  the 
development  of  a  source  selection  handbook  including  an 
explanation  of  the  process,  detailed  instructions ,  and 
sample  documents.  Such  an  effort  could  study  various 
source  selections  to  develop  instructions  and  identify 
the  best  documentation  which  could  then  be  tailored  by 
source  selection  teams. 

The  final  recommendation  for  additional  research 
involves  the  development  of  a  training  course  on  source 
selection.  The  course  should  include  an  overview  to 
provide  an  understanding  of  the  process,  instructional 
sessions  on  specific  tasks,  and  multi-disciplinary 
sessions  to  include  activities  such  as  case  studies  and 
exercises  in  planning,  strategy  formulation,  problem¬ 
solving,  and  document  preparation  and  review.  In 
addition  t<^  developing  a  thorough  understanding  of  the 
source  selection  process,  the  objectives  of  the  course 
should  include  providing  practical  experience  with 
skills  required  in  source  selection  and  building  the 
teamwork  which  is  essential  for  effective  conduct  of 
source  selections. 
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Appendix  A:  InttrvUw  Questions 


Nam* :  _  Interview  No. _ 

Position;  _  Warranted?  Y  N 


Product  Division:  _ 

Programts)  :  _ _ _ _ _ 

No.  years  buyer/PCO  experience:  _ 

Oth«r  non-buyer/PCO  acquisition-related  experience: _ 


Experience  at  other  product  divisions: 


Extent  of  source  selection  experience: 


123 


PRE-EVALUATION  ACTIVITIES 

3u3iness  Strategy  Panel/Acqui3ition  Strategy  Panel 
Identification  of  sources 

Market  survey/synopsis 
Source  Selection  Plan 
Preparation 
Review  and  approval 
Evaluation  criteria  development 
Cost 

Speci f ic 
Assessment 

General  considerations 
Evaluation  standards 
Instructions  to  Offerors  preparation 
Request  for  Proposal  issuance 
PR  package  development 
RFP  preparation 
Review 

PROPOSAL  EVALUATION  AND  CONTRACT  AWARD 
Receipt  of  proposals 
Quick  Look  briefing  or  memo 
Oral  briefings  by  offorors  (optional) 

Technical  evaluation 

Deficiency  Reports 
Clarification  Requests 
Modification  Requests 
Technical  evaluation  reports 
Performance  Risk  Analysis 
Mid-Term  briefing 

Co3t/price  analysis/Mos l  Probable  Cost 
Competitive  range  determination 
Model  contracts 
Preparation 
Review 
Discussions 

Best  and  Final  Offers  BAFO) 

Request  for  BAFO 
Receipt  of  3AF0 
Technical  evaluation  update 
SSEB  Executive  Summary  Report 

SSEB  briefing  to  SSAC  or  SSET  Proposal  Analysis 

SSAC  Analysis  Report  Report 

Final  SSA  briefing 

SSA  decision  document 

Contract  execution 

Contract  review 
Contract  approval 
Contract  distribution 
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Are  there  any  pre-evaluation  activities  on  this  li3t 
with  which  you  have  no  experience’ 

What  is  the  first  item  on  the  li3t  which  you  would 
characterise  as  a  problem  in  the  source  selection 
process’ 

Do  you  consider  this  area  to  be  a  minor,  moderate,  or 
major  problem’ 

What  are  the  problems  that  you've  experienced  with  this 
activity’ 

What  do  you  think  caused  the  prcblem(s)’ 

What  actions  were  taken  to  solve  the  problem(s)’ 

Do  you  have  any  suggestions  for  avoiding  the  problom(s) 
m  the  future’ 

(This  series  of  questions  is  repeated  for  every  activity 
identified  as  representing  a  problem  in  the  source 
selection  process) . 

Do  you  see  any  activities  listed  which  require  an 
investment  of  time  and/or  energy  m  excess  of  the 
benefit  that  results’ 

Could  this  activity  bo  deleted  without  detriment  to  the 
process,  or  is  there  a  more  cost  effective  way  to 
accomplish  the  effort’  IRepeated  fo"  each  activity 
identi f led J . 

Have  you  ever  used  any  unusual  means  to  solve  problems, 
avoid  problems  you've  experienced  on  past  source 
selections,  or  to  accelerate  a  source  selection 
schedule’ 

How  well  did  this  initiative  work’ 

Would  you  use  the  same  technique  again’ 

Have  you  thought  of  any  other  techniques  that  you  would 
like  to  try  if  you  ran  into  the  same  situation  again’ 

Do  you  have  any  comments  about  the  source  selection 
process  or  this  interview’ 
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Appendix  B:  Interview  Results 
Problem  Areas 


Mo 

Ml  n- 

Mod- 

Total 

Activity 

Exp 

Min 

Mod 

Mod 

Maj 

Maj 

Prob 

BSP/AS? 

O 

3 

1 

4 

Source  ID 

A 

1 

5 

SS? 

1 

1 

Prep 
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1 

1 

o 

mm 

Review 

1 

App'  1 

o 

mm 

Criteria 

2 

3 

3 

6 

Standards 

1 

4 

4 

I  TO 

1 

2 

n 

mm 

4 

RFP 

PR  pkg 

2 

o 

mt 

4 

8 

RFP  prep 

1 

1 

2 

Review 

3 

2 

5 

Proposals 

1 

1 

Quick  Look 

1 

1 

Oral  brief. 

2 

0 

Tech  eval 

2 

2 

4 

DRs 

1 

1 

1 

3 

6 

CRs 

1 

1 

1 

2 

5 

MRs 

1 

1 

2 

Reports 

1 

1 

PRA 

1 

1 

Mid-Term 

1 

2 

3 

Cost/price/ 

MPC 

1 

1 

3 

4 

Competi tive 
range 

0 

Modal  k  prep 

1 

1 

1 

3 

6 

Discussions 

1 

1 
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RF?  prep  tcont .  i 


RFP  review  4 


Quick  Look 


Tech  eval 


4 


Mid-Term  4 

MFC  2 

Cost/price  3 

analysis 


Model  contract  1 
preparation 


-  RFP  preparation  can  start 
before  complete  PR  package 
is  received;  develop  PR 
package  and  RF?  in  parallel 

-  too  many  higher  levol 
reviews:  ART  team,  ASD/PMC, 
and  AFSC 

-  Delete  AFSC  reviews;  they 
are  not  part  of  the  ART  and 
do  not  understand  the 
problems;  their  demands 
cause  MRs ;  they  should 
either  be  totally  involved 
in  the  program  or  not  be 
involved  at  all 

-  make  the  ART  the  sole  RFP 
review  at  ASD 

-  combine  with  Mid-Term 

-  delete 

-  involves  too  many  people 
with  too  little  skill 

-  CR/DR  approval  by  SSA  wastes 
time;  cannot  issue  CRs/DRs 
until  after  Mid-Term 

-  combine  or  reduce  number  of 
evaluation  reports;  time- 
consuming  and  redundant 

-  delete  S5AC  Analysis  Report 

-  delete  (3  responses) 

make  optional  or  allow  memo 

-  delete  (2  responses) 

-  audits  useless  due  to  method 
of  price  determination  in 
source  selection;  delete 

-  delete 

-  limit  cost  data  required 

-  tune  DCAS  and  DCAA  to  new 
accelerated  schedules 

-  ensure  adequate  contracts 

team:  dedicated  CO,  two 

buyers,  clerk 
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Discuss  ions 


2 


3AF0s 


SSA  briefing  1 


Contract  review  1 
and  approval 


-  use  more  oral  discussion  and 
fewer  CRs  and  DRs 

-  award  without  discussions 
more  often  when  appropriate 

-  revise  criteria  to  allow 
more  awards  without 
discussions 

-  use  less  frequently 

-  when  SAF  is  SSA,  briefing  is 
very  hard  to  schedule;  allow 
use  of  report  in  lieu  of 
brief lng 

-  raise  manual  approval 
thresholds 

-  process  takes  too  long; 
should  be  much  faster 
because  final  contract  has 
been  through  numerous 
reviews 
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